Jul 29 03:55:33.829092 ip-10-0-135-138 systemd[1]: Starting Kubernetes Kubelet...
Jul 29 03:55:34.233342 ip-10-0-135-138 kubenswrapper[2570]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 03:55:34.233342 ip-10-0-135-138 kubenswrapper[2570]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 29 03:55:34.233342 ip-10-0-135-138 kubenswrapper[2570]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 03:55:34.233342 ip-10-0-135-138 kubenswrapper[2570]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 29 03:55:34.233342 ip-10-0-135-138 kubenswrapper[2570]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 03:55:34.234038 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.233392    2570 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 29 03:55:34.237672 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237657    2570 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 03:55:34.237672 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237672    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 03:55:34.237738 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237678    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 03:55:34.237738 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237683    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 03:55:34.237738 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237687    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 03:55:34.237738 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237690    2570 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 03:55:34.237738 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237693    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 03:55:34.237738 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237695    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 03:55:34.237738 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237698    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 03:55:34.237738 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237700    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 03:55:34.237738 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237703    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 03:55:34.237738 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237709    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 03:55:34.237738 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237712    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 03:55:34.237738 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237714    2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 03:55:34.237738 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237717    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 03:55:34.237738 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237719    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 03:55:34.237738 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237722    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 03:55:34.237738 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237725    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 03:55:34.237738 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237727    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 03:55:34.237738 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237730    2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 03:55:34.237738 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237732    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 03:55:34.237738 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237734    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 03:55:34.238229 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237737    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 03:55:34.238229 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237739    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 03:55:34.238229 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237744    2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 03:55:34.238229 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237747    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 03:55:34.238229 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237751    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 03:55:34.238229 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237754    2570 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 03:55:34.238229 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237757    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 03:55:34.238229 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237759    2570 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 03:55:34.238229 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237762    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 03:55:34.238229 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237765    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 03:55:34.238229 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237767    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 03:55:34.238229 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237769    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 03:55:34.238229 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237772    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 03:55:34.238229 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237774    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 03:55:34.238229 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237776    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 03:55:34.238229 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237779    2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 03:55:34.238229 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237781    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 03:55:34.238229 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237783    2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 03:55:34.238229 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237786    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 03:55:34.238685 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237788    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 03:55:34.238685 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237791    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 03:55:34.238685 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237793    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 03:55:34.238685 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237796    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 03:55:34.238685 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237799    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 03:55:34.238685 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237801    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 03:55:34.238685 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237819    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 03:55:34.238685 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237822    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 03:55:34.238685 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237824    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 03:55:34.238685 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237827    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 03:55:34.238685 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237829    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 03:55:34.238685 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237834    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 03:55:34.238685 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237837    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 03:55:34.238685 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237839    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 03:55:34.238685 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237843    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 03:55:34.238685 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237845    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 03:55:34.238685 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237848    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 03:55:34.238685 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237850    2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 03:55:34.238685 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237853    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 03:55:34.238685 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237855    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 03:55:34.239189 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237858    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 03:55:34.239189 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237860    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 03:55:34.239189 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237863    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 03:55:34.239189 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237865    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 03:55:34.239189 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237867    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 03:55:34.239189 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237870    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 03:55:34.239189 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237872    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 03:55:34.239189 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237874    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 03:55:34.239189 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237877    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 03:55:34.239189 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237879    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 03:55:34.239189 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237882    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 03:55:34.239189 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237884    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 03:55:34.239189 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237887    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 03:55:34.239189 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237889    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 03:55:34.239189 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237892    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 03:55:34.239189 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237894    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 03:55:34.239189 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237898    2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 03:55:34.239189 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237900    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 03:55:34.239189 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237902    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 03:55:34.239189 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237905    2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 03:55:34.239754 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237907    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 03:55:34.239754 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237910    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 03:55:34.239754 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237913    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 03:55:34.239754 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237915    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 03:55:34.239754 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237917    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 03:55:34.239754 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237920    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 03:55:34.239754 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237924    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 03:55:34.239754 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237927    2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 03:55:34.239754 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237930    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 03:55:34.239754 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237932    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 03:55:34.239754 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237935    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 03:55:34.239754 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237937    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 03:55:34.239754 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237939    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 03:55:34.239754 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237942    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 03:55:34.239754 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237944    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 03:55:34.239754 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237947    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 03:55:34.239754 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237949    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 03:55:34.239754 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.237952    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 03:55:34.239754 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238633    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 03:55:34.239754 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238641    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 03:55:34.240515 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238644    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 03:55:34.240515 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238646    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 03:55:34.240515 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238650    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 03:55:34.240515 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238652    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 03:55:34.240515 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238655    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 03:55:34.240515 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238658    2570 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 03:55:34.240515 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238660    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 03:55:34.240515 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238662    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 03:55:34.240515 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238665    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 03:55:34.240515 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238671    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 03:55:34.240515 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238674    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 03:55:34.240515 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238677    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 03:55:34.240515 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238679    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 03:55:34.240515 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238682    2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 03:55:34.240515 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238684    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 03:55:34.240515 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238689    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 03:55:34.240515 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238693    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 03:55:34.240515 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238696    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 03:55:34.240515 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238699    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 03:55:34.241295 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238702    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 03:55:34.241295 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238705    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 03:55:34.241295 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238708    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 03:55:34.241295 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238710    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 03:55:34.241295 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238713    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 03:55:34.241295 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238716    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 03:55:34.241295 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238719    2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 03:55:34.241295 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238721    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 03:55:34.241295 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238724    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 03:55:34.241295 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238726    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 03:55:34.241295 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238729    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 03:55:34.241295 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238731    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 03:55:34.241295 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238734    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 03:55:34.241295 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238737    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 03:55:34.241295 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238740    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 03:55:34.241295 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238743    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 03:55:34.241295 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238746    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 03:55:34.241295 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238748    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 03:55:34.241295 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238751    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 03:55:34.241884 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238753    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 03:55:34.241884 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238756    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 03:55:34.241884 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238758    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 03:55:34.241884 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238760    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 03:55:34.241884 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238763    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 03:55:34.241884 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238765    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 03:55:34.241884 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238768    2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 03:55:34.241884 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238770    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 03:55:34.241884 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238773    2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 03:55:34.241884 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238776    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 03:55:34.241884 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238778    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 03:55:34.241884 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238781    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 03:55:34.241884 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238784    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 03:55:34.241884 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238786    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 03:55:34.241884 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238789    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 03:55:34.241884 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238791    2570 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 03:55:34.241884 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238794    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 03:55:34.241884 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238796    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 03:55:34.241884 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238799    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 03:55:34.242442 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238801    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 03:55:34.242442 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238803    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 03:55:34.242442 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238820    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 03:55:34.242442 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238822    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 03:55:34.242442 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238825    2570 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 03:55:34.242442 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238827    2570 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 03:55:34.242442 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238830    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 03:55:34.242442 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238832    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 03:55:34.242442 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238835    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 03:55:34.242442 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238838    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 03:55:34.242442 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238840    2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 03:55:34.242442 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238844    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 03:55:34.242442 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238846    2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 03:55:34.242442 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238849    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 03:55:34.242442 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238851    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 03:55:34.242442 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238854    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 03:55:34.242442 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238856    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 03:55:34.242442 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238859    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 03:55:34.242442 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238862    2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 03:55:34.242442 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238864    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 03:55:34.243174 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238867    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 03:55:34.243174 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238869    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 03:55:34.243174 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238872    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 03:55:34.243174 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238874    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 03:55:34.243174 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238877    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 03:55:34.243174 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238880    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 03:55:34.243174 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238883    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 03:55:34.243174 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238886    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 03:55:34.243174 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238888    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 03:55:34.243174 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238890    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 03:55:34.243174 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238893    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 03:55:34.243174 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238896    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 03:55:34.243174 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238898    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 03:55:34.243174 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238900    2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 03:55:34.243174 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238903    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 03:55:34.243174 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238905    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 03:55:34.243174 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238907    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 03:55:34.243174 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238911    2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 03:55:34.243174 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238915    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 03:55:34.243174 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.238917    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 03:55:34.243686 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239275    2570 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 03:55:34.243686 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239280    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 03:55:34.243686 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239283    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 03:55:34.243686 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239286    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 03:55:34.243686 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239289    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 03:55:34.243686 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239291    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 03:55:34.243686 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239294    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 03:55:34.243686 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239297    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 03:55:34.243686 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239299    2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 03:55:34.243686 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239302    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 03:55:34.243686 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239304    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 03:55:34.243686 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239307    2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 03:55:34.243686 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239309    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 03:55:34.243686 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239312    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 03:55:34.243686 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239314    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 03:55:34.243686 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239316    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 03:55:34.243686 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239319    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 03:55:34.243686 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239322    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 03:55:34.243686 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239325    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 03:55:34.243686 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239327    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 03:55:34.244264 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239330    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 03:55:34.244264 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239332    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 03:55:34.244264 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239335    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 03:55:34.244264 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239338    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 03:55:34.244264 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239340    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 03:55:34.244264 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239343    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 03:55:34.244264 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239345    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 03:55:34.244264 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239348    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 03:55:34.244264 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239351    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 03:55:34.244264 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239353    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 03:55:34.244264 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239356    2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 03:55:34.244264 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239358    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 03:55:34.244264 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239361    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 03:55:34.244264 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239363    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 03:55:34.244264 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239366    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 03:55:34.244264 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239368    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 03:55:34.244264 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239370    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 03:55:34.244264 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239373    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 03:55:34.244264 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239375    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 03:55:34.244264 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239378    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 03:55:34.244763 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239381    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 03:55:34.244763 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239385    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 03:55:34.244763 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239388    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 03:55:34.244763 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239391    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 03:55:34.244763 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239393    2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 03:55:34.244763 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239396    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 03:55:34.244763 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239398    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 03:55:34.244763 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239400    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 03:55:34.244763 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239403    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 03:55:34.244763 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239406    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 03:55:34.244763 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239409    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 03:55:34.244763 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239411    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 03:55:34.244763 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239414    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 03:55:34.244763 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239417    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 03:55:34.244763 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239419    2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 03:55:34.244763 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239421    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 03:55:34.244763 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239424    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 03:55:34.244763 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239426    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 03:55:34.244763 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239429    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 03:55:34.245245 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239431    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 03:55:34.245245 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239433    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 03:55:34.245245 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239436    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 03:55:34.245245 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239438    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 03:55:34.245245 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239440    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 03:55:34.245245 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239443    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 03:55:34.245245 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239445    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 03:55:34.245245 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239448    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 03:55:34.245245 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239450    2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 03:55:34.245245 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239452    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 03:55:34.245245 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239455    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 03:55:34.245245 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239457    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 03:55:34.245245 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239460    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 03:55:34.245245 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239462    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 03:55:34.245245 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239464    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 03:55:34.245245 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239467    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 03:55:34.245245 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239471    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 03:55:34.245245 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239477    2570 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 03:55:34.245245 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239480    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 03:55:34.245718 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239482    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 03:55:34.245718 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239485    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 03:55:34.245718 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239489    2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 03:55:34.245718 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239492    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 03:55:34.245718 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239495    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 03:55:34.245718 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239498    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 03:55:34.245718 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239502    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 03:55:34.245718 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239505    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 03:55:34.245718 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239508    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 03:55:34.245718 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239510    2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 03:55:34.245718 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239513    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 03:55:34.245718 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239516    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 03:55:34.245718 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239518    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 03:55:34.245718 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239520    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 03:55:34.245718 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239523    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 03:55:34.245718 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239525    2570 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 03:55:34.245718 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239528    2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 03:55:34.245718 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239530    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 03:55:34.245718 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239533    2570 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 03:55:34.245718 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239535    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 03:55:34.246260 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.239538    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 03:55:34.246260 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.239605    2570 flags.go:64] FLAG: --address="0.0.0.0"
Jul 29 03:55:34.246260 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.239612    2570 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 29 03:55:34.246260 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.239618    2570 flags.go:64] FLAG: --anonymous-auth="true"
Jul 29 03:55:34.246260 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.239622    2570 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 29 03:55:34.246260 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.239626    2570 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 29 03:55:34.246260 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.239629    2570 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 29 03:55:34.246260 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.239633    2570 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 29 03:55:34.246260 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.239636    2570 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 29 03:55:34.246260 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.239639    2570 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 29 03:55:34.246260 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.239642    2570 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 29 03:55:34.246260 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.239646    2570 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 29 03:55:34.246260 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.239649    2570 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 29 03:55:34.246260 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.239653    2570 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 29 03:55:34.246260 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.239656    2570 flags.go:64] FLAG: --cgroup-root=""
Jul 29 03:55:34.246260 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.239658    2570 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 29 03:55:34.246260 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.239662    2570 flags.go:64] FLAG: --client-ca-file=""
Jul 29 03:55:34.246260 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.239664    2570 flags.go:64] FLAG: --cloud-provider="external"
Jul 29 03:55:34.246260 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.239667    2570 flags.go:64] FLAG: --cluster-dns="[]"
Jul 29 03:55:34.246260 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.239671    2570 flags.go:64] FLAG: --cluster-domain=""
Jul 29 03:55:34.246260 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.239674    2570 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 29 03:55:34.246260 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.239677    2570 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 29 03:55:34.246260 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.239680    2570 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 29 03:55:34.246260 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.239683    2570 flags.go:64] FLAG: --container-log-max-files="5"
Jul 29 03:55:34.246867 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.239687    2570 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 29 03:55:34.246867 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.239690    2570 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 29 03:55:34.246867 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.239692    2570 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 29 03:55:34.246867 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.239696    2570 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 29 03:55:34.246867 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.239698    2570 flags.go:64] FLAG: --contention-profiling="false"
Jul 29 03:55:34.246867 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.239701    2570 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 29 03:55:34.246867 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.239704    2570 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 29 03:55:34.246867 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.239707    2570 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 29 03:55:34.246867 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.239710    2570 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 29 03:55:34.246867 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.239714    2570 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 29 03:55:34.246867 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.239717    2570 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 29 03:55:34.246867 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.239719    2570 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 29 03:55:34.246867 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.239722    2570 flags.go:64] FLAG: --enable-load-reader="false"
Jul 29 03:55:34.246867 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.239725    2570 flags.go:64] FLAG: --enable-server="true"
Jul 29 03:55:34.246867 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.239727    2570 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 29 03:55:34.246867 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.239731    2570 flags.go:64] FLAG: --event-burst="100"
Jul 29 03:55:34.246867 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.239734    2570 flags.go:64] FLAG: --event-qps="50"
Jul 29 03:55:34.246867 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.239737    2570 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 29 03:55:34.246867 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.239740    2570 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 29 03:55:34.246867 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.239743    2570 flags.go:64] FLAG: --eviction-hard=""
Jul 29 03:55:34.246867 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.239747    2570 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 29 03:55:34.246867 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.239752    2570 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 29 03:55:34.246867 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.239755    2570 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 29 03:55:34.246867 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.239758    2570 flags.go:64] FLAG: --eviction-soft=""
Jul 29 03:55:34.246867 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.239761    2570 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 29 03:55:34.247532 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.239764    2570 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 29 03:55:34.247532 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.239767    2570 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 29 03:55:34.247532 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.239770    2570 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 29 03:55:34.247532 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.239773    2570 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 29 03:55:34.247532 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.239776    2570 flags.go:64] FLAG: --fail-swap-on="true"
Jul 29 03:55:34.247532 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.239778    2570 flags.go:64] FLAG: --feature-gates=""
Jul 29 03:55:34.247532 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.239782    2570 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 29 03:55:34.247532 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.239785    2570 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 29 03:55:34.247532 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.239787    2570 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 29 03:55:34.247532 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.239790    2570 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 29 03:55:34.247532 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.239793    2570 flags.go:64] FLAG: --healthz-port="10248"
Jul 29 03:55:34.247532 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.239795    2570 flags.go:64] FLAG: --help="false"
Jul 29 03:55:34.247532 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.239798    2570 flags.go:64] FLAG: --hostname-override="ip-10-0-135-138.ec2.internal"
Jul 29 03:55:34.247532 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.239801    2570 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 29 03:55:34.247532 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.239870    2570 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 29 03:55:34.247532 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.239876    2570 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 29 03:55:34.247532 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.239880    2570 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 29 03:55:34.247532 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.239884    2570 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 29 03:55:34.247532 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.239887    2570 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 29 03:55:34.247532 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240114    2570 flags.go:64] FLAG: --image-service-endpoint=""
Jul 29 03:55:34.247532 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240124    2570 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 29 03:55:34.247532 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240131    2570 flags.go:64] FLAG: --kube-api-burst="100"
Jul 29 03:55:34.247532 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240137    2570 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 29 03:55:34.248141 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240143    2570 flags.go:64] FLAG: --kube-api-qps="50"
Jul 29 03:55:34.248141 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240149    2570 flags.go:64] FLAG: --kube-reserved=""
Jul 29 03:55:34.248141 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240155    2570 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 29 03:55:34.248141 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240160    2570 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 29 03:55:34.248141 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240171    2570 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 29 03:55:34.248141 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240176    2570 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 29 03:55:34.248141 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240180    2570 flags.go:64] FLAG: --lock-file=""
Jul 29 03:55:34.248141 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240185    2570 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 29 03:55:34.248141 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240190    2570 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 29 03:55:34.248141 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240195    2570 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 29 03:55:34.248141 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240204    2570 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 29 03:55:34.248141 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240209    2570 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 29 03:55:34.248141 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240218    2570 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 29 03:55:34.248141 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240223    2570 flags.go:64] FLAG: --logging-format="text"
Jul 29 03:55:34.248141 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240228    2570 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 29 03:55:34.248141 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240233    2570 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 29 03:55:34.248141 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240238    2570 flags.go:64] FLAG: --manifest-url=""
Jul 29 03:55:34.248141 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240243    2570 flags.go:64] FLAG: --manifest-url-header=""
Jul 29 03:55:34.248141 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240249    2570 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 29 03:55:34.248141 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240254    2570 flags.go:64] FLAG: --max-open-files="1000000"
Jul 29 03:55:34.248141 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240265    2570 flags.go:64] FLAG: --max-pods="110"
Jul 29 03:55:34.248141 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240270    2570 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 29 03:55:34.248141 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240275    2570 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 29 03:55:34.248141 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240279    2570 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 29 03:55:34.248141 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240285    2570 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 29 03:55:34.248764 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240290    2570 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 29 03:55:34.248764 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240297    2570 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 29 03:55:34.248764 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240302    2570 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 29 03:55:34.248764 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240327    2570 flags.go:64] FLAG: --node-status-max-images="50"
Jul 29 03:55:34.248764 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240333    2570 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 29 03:55:34.248764 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240338    2570 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 29 03:55:34.248764 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240343    2570 flags.go:64] FLAG: --pod-cidr=""
Jul 29 03:55:34.248764 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240347    2570 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 29 03:55:34.248764 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240357    2570 flags.go:64] FLAG: --pod-manifest-path=""
Jul 29 03:55:34.248764 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240362    2570 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 29 03:55:34.248764 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240366    2570 flags.go:64] FLAG: --pods-per-core="0"
Jul 29 03:55:34.248764 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240376    2570 flags.go:64] FLAG: --port="10250"
Jul 29 03:55:34.248764 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240380    2570 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 29 03:55:34.248764 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240385    2570 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0176a64014479be37"
Jul 29 03:55:34.248764 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240390    2570 flags.go:64] FLAG: --qos-reserved=""
Jul 29 03:55:34.248764 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240395    2570 flags.go:64] FLAG: --read-only-port="10255"
Jul 29 03:55:34.248764 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240400    2570 flags.go:64] FLAG: --register-node="true"
Jul 29 03:55:34.248764 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240405    2570 flags.go:64] FLAG: --register-with-taints=""
Jul 29 03:55:34.248764 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240411    2570 flags.go:64] FLAG: --registry-burst="10"
Jul 29 03:55:34.248764 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240415    2570 flags.go:64] FLAG: --registry-qps="5"
Jul 29 03:55:34.248764 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240424    2570 flags.go:64] FLAG: --reserved-cpus=""
Jul 29 03:55:34.248764 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240428    2570 flags.go:64] FLAG: --reserved-memory=""
Jul 29 03:55:34.248764 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240434    2570 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 29 03:55:34.248764 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240440    2570 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 29 03:55:34.248764 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240445    2570 flags.go:64] FLAG: --rotate-certificates="false"
Jul 29 03:55:34.249391 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240449    2570 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 29 03:55:34.249391 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240454    2570 flags.go:64] FLAG: --runonce="false"
Jul 29 03:55:34.249391 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240458    2570 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 29 03:55:34.249391 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240468    2570 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 29 03:55:34.249391 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240472    2570 flags.go:64] FLAG: --seccomp-default="false"
Jul 29 03:55:34.249391 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240477    2570 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 29 03:55:34.249391 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240482    2570 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 29 03:55:34.249391 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240487    2570 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 29 03:55:34.249391 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240491    2570 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 29 03:55:34.249391 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240498    2570 flags.go:64] FLAG: --storage-driver-password="root"
Jul 29 03:55:34.249391 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240503    2570 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 29 03:55:34.249391 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240508    2570 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 29 03:55:34.249391 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240518    2570 flags.go:64] FLAG: --storage-driver-user="root"
Jul 29 03:55:34.249391 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240523    2570 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 29 03:55:34.249391 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240528    2570 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 29 03:55:34.249391 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240533    2570 flags.go:64] FLAG: --system-cgroups=""
Jul 29 03:55:34.249391 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240538    2570 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 29 03:55:34.249391 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240547    2570 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 29 03:55:34.249391 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240557    2570 flags.go:64] FLAG: --tls-cert-file=""
Jul 29 03:55:34.249391 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240562    2570 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 29 03:55:34.249391 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240570    2570 flags.go:64] FLAG: --tls-min-version=""
Jul 29 03:55:34.249391 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240574    2570 flags.go:64] FLAG: --tls-private-key-file=""
Jul 29 03:55:34.249391 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240579    2570 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 29 03:55:34.249391 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240584    2570 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 29 03:55:34.249391 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240588    2570 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 29 03:55:34.250049 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240593    2570 flags.go:64] FLAG: --v="2"
Jul 29 03:55:34.250049 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240599    2570 flags.go:64] FLAG: --version="false"
Jul 29 03:55:34.250049 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240611    2570 flags.go:64] FLAG: --vmodule=""
Jul 29 03:55:34.250049 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240616    2570 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 29 03:55:34.250049 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.240622    2570 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 29 03:55:34.250049 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241065    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 03:55:34.250049 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241081    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 03:55:34.250049 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241086    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 03:55:34.250049 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241092    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 03:55:34.250049 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241097    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 03:55:34.250049 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241102    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 03:55:34.250049 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241107    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 03:55:34.250049 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241111    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 03:55:34.250049 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241115    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 03:55:34.250049 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241119    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 03:55:34.250049 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241123    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 03:55:34.250049 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241127    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 03:55:34.250049 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241131    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 03:55:34.250049 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241136    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 03:55:34.250049 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241140    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 03:55:34.250049 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241144    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 03:55:34.250561 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241148    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 03:55:34.250561 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241153    2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 03:55:34.250561 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241157    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 03:55:34.250561 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241161    2570 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 03:55:34.250561 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241164    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 03:55:34.250561 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241169    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 03:55:34.250561 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241174    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 03:55:34.250561 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241178    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 03:55:34.250561 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241182    2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 03:55:34.250561 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241186    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 03:55:34.250561 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241191    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 03:55:34.250561 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241195    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 03:55:34.250561 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241200    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 03:55:34.250561 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241204    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 03:55:34.250561 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241209    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 03:55:34.250561 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241213    2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 03:55:34.250561 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241217    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 03:55:34.250561 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241222    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 03:55:34.250561 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241227    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 03:55:34.250561 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241232    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 03:55:34.251175 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241236    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 03:55:34.251175 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241240    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 03:55:34.251175 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241244    2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 03:55:34.251175 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241248    2570 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 03:55:34.251175 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241252    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 03:55:34.251175 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241256    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 03:55:34.251175 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241260    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 03:55:34.251175 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241267    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 03:55:34.251175 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241275    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 03:55:34.251175 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241279    2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 03:55:34.251175 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241284    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 03:55:34.251175 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241289    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 03:55:34.251175 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241292    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 03:55:34.251175 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241296    2570 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 03:55:34.251175 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241302    2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 03:55:34.251175 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241307    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 03:55:34.251175 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241312    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 03:55:34.251175 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241316    2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 03:55:34.251175 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241321    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 03:55:34.251629 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241325    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 03:55:34.251629 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241330    2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 03:55:34.251629 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241334    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 03:55:34.251629 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241341    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 03:55:34.251629 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241347    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 03:55:34.251629 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241351    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 03:55:34.251629 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241355    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 03:55:34.251629 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241359    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 03:55:34.251629 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241363    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 03:55:34.251629 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241367    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 03:55:34.251629 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241372    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 03:55:34.251629 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241377    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 03:55:34.251629 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241381    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 03:55:34.251629 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241386    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 03:55:34.251629 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241390    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 03:55:34.251629 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241394    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 03:55:34.251629 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241398    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 03:55:34.251629 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241402    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 03:55:34.251629 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241406    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 03:55:34.252105 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241410    2570 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 03:55:34.252105 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241415    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 03:55:34.252105 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241419    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 03:55:34.252105 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241424    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 03:55:34.252105 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241427    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 03:55:34.252105 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241432    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 03:55:34.252105 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241436    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 03:55:34.252105 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241440    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 03:55:34.252105 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241444    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 03:55:34.252105 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241448    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 03:55:34.252105 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241452    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 03:55:34.252105 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241456    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 03:55:34.252105 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241460    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 03:55:34.252105 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241465    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 03:55:34.252105 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241469    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 03:55:34.252105 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241473    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 03:55:34.252105 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241477    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 03:55:34.252105 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241481    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 03:55:34.252105 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241485    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 03:55:34.252105 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241489    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 03:55:34.252595 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241493    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 03:55:34.252595 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241497    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 03:55:34.252595 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241502    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 03:55:34.252595 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241507    2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 03:55:34.252595 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.241511    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 03:55:34.252595 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.242071    2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 03:55:34.252595 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.248677    2570 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 29 03:55:34.252595 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.248691    2570 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 29 03:55:34.252595 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.249600    2570 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 29 03:55:34.252595 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.249607    2570 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 29 03:55:34.252595 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249666    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 03:55:34.252595 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249671    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 03:55:34.252595 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249675    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 03:55:34.252595 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249678    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 03:55:34.252595 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249681    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 03:55:34.252993 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249683    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 03:55:34.252993 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249686    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 03:55:34.252993 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249689    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 03:55:34.252993 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249691    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 03:55:34.252993 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249694    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 03:55:34.252993 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249696    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 03:55:34.252993 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249698    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 03:55:34.252993 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249701    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 03:55:34.252993 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249703    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 03:55:34.252993 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249706    2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 03:55:34.252993 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249709    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 03:55:34.252993 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249712    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 03:55:34.252993 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249714    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 03:55:34.252993 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249718    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 03:55:34.252993 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249723    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 03:55:34.252993 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249726    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 03:55:34.252993 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249728    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 03:55:34.252993 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249731    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 03:55:34.252993 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249733    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 03:55:34.252993 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249736    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 03:55:34.253500 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249738    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 03:55:34.253500 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249741    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 03:55:34.253500 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249743    2570 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 03:55:34.253500 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249746    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 03:55:34.253500 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249749    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 03:55:34.253500 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249751    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 03:55:34.253500 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249754    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 03:55:34.253500 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249757    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 03:55:34.253500 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249759    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 03:55:34.253500 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249761    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 03:55:34.253500 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249764    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 03:55:34.253500 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249767    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 03:55:34.253500 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249769    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 03:55:34.253500 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249772    2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 03:55:34.253500 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249774    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 03:55:34.253500 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249776    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 03:55:34.253500 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249779    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 03:55:34.253500 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249781    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 03:55:34.253500 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249783    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 03:55:34.253500 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249786    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 03:55:34.254006 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249788    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 03:55:34.254006 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249791    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 03:55:34.254006 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249794    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 03:55:34.254006 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249796    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 03:55:34.254006 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249798    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 03:55:34.254006 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249801    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 03:55:34.254006 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249815    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 03:55:34.254006 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249819    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 03:55:34.254006 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249821    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 03:55:34.254006 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249824    2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 03:55:34.254006 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249827    2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 03:55:34.254006 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249829    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 03:55:34.254006 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249832    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 03:55:34.254006 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249834    2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 03:55:34.254006 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249837    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 03:55:34.254006 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249840    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 03:55:34.254006 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249844    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 03:55:34.254006 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249848    2570 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 03:55:34.254006 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249851    2570 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 03:55:34.254465 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249854    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 03:55:34.254465 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249856    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 03:55:34.254465 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249859    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 03:55:34.254465 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249861    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 03:55:34.254465 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249864    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 03:55:34.254465 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249866    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 03:55:34.254465 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249869    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 03:55:34.254465 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249871    2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 03:55:34.254465 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249874    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 03:55:34.254465 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249876    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 03:55:34.254465 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249879    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 03:55:34.254465 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249881    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 03:55:34.254465 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249883    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 03:55:34.254465 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249886    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 03:55:34.254465 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249889    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 03:55:34.254465 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249892    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 03:55:34.254465 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249894    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 03:55:34.254465 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249897    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 03:55:34.254465 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249899    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 03:55:34.254959 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249901    2570 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 03:55:34.254959 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249904    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 03:55:34.254959 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249909    2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 03:55:34.254959 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249912    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 03:55:34.254959 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249915    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 03:55:34.254959 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249917    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 03:55:34.254959 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249920    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 03:55:34.254959 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249923    2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 03:55:34.254959 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249927    2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 03:55:34.254959 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249929    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 03:55:34.254959 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249932    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 03:55:34.254959 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249935    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 03:55:34.254959 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249937    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 03:55:34.254959 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249940    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 03:55:34.254959 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249942    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 03:55:34.254959 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.249945    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 03:55:34.255343 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.249950    2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 03:55:34.255343 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250055    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 03:55:34.255343 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250060    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 03:55:34.255343 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250064    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 03:55:34.255343 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250066    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 03:55:34.255343 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250070    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 03:55:34.255343 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250073    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 03:55:34.255343 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250076    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 03:55:34.255343 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250078    2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 03:55:34.255343 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250081    2570 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 03:55:34.255343 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250083    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 03:55:34.255343 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250086    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 03:55:34.255343 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250089    2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 03:55:34.255343 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250092    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 03:55:34.255343 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250095    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 03:55:34.255719 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250097    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 03:55:34.255719 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250100    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 03:55:34.255719 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250102    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 03:55:34.255719 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250105    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 03:55:34.255719 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250108    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 03:55:34.255719 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250110    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 03:55:34.255719 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250112    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 03:55:34.255719 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250115    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 03:55:34.255719 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250117    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 03:55:34.255719 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250120    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 03:55:34.255719 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250122    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 03:55:34.255719 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250125    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 03:55:34.255719 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250129    2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 03:55:34.255719 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250132    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 03:55:34.255719 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250135    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 03:55:34.255719 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250137    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 03:55:34.255719 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250140    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 03:55:34.255719 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250143    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 03:55:34.255719 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250145    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 03:55:34.255719 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250148    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 03:55:34.256222 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250150    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 03:55:34.256222 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250153    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 03:55:34.256222 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250156    2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 03:55:34.256222 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250158    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 03:55:34.256222 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250160    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 03:55:34.256222 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250163    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 03:55:34.256222 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250165    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 03:55:34.256222 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250167    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 03:55:34.256222 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250170    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 03:55:34.256222 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250173    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 03:55:34.256222 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250175    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 03:55:34.256222 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250178    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 03:55:34.256222 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250180    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 03:55:34.256222 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250182    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 03:55:34.256222 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250185    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 03:55:34.256222 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250187    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 03:55:34.256222 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250190    2570 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 03:55:34.256222 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250193    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 03:55:34.256222 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250195    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 03:55:34.256222 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250197    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 03:55:34.256697 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250200    2570 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 03:55:34.256697 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250202    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 03:55:34.256697 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250205    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 03:55:34.256697 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250208    2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 03:55:34.256697 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250210    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 03:55:34.256697 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250213    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 03:55:34.256697 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250215    2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 03:55:34.256697 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250217    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 03:55:34.256697 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250220    2570 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 03:55:34.256697 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250222    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 03:55:34.256697 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250225    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 03:55:34.256697 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250228    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 03:55:34.256697 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250232    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 03:55:34.256697 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250234    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 03:55:34.256697 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250237    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 03:55:34.256697 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250239    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 03:55:34.256697 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250242    2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 03:55:34.256697 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250244    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 03:55:34.256697 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250246    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 03:55:34.256697 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250249    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 03:55:34.257246 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250251    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 03:55:34.257246 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250254    2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 03:55:34.257246 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250256    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 03:55:34.257246 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250259    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 03:55:34.257246 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250261    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 03:55:34.257246 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250263    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 03:55:34.257246 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250266    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 03:55:34.257246 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250268    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 03:55:34.257246 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250271    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 03:55:34.257246 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250273    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 03:55:34.257246 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250276    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 03:55:34.257246 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250279    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 03:55:34.257246 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250281    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 03:55:34.257246 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250284    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 03:55:34.257246 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250286    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 03:55:34.257246 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250289    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 03:55:34.257246 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250291    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 03:55:34.257246 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250295    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 03:55:34.257688 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250298    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 03:55:34.257688 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250301    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 03:55:34.257688 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250304    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 03:55:34.257688 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250306    2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 03:55:34.257688 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250309    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 03:55:34.257688 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250312    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 03:55:34.257688 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:34.250315    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 03:55:34.257688 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.250319    2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 03:55:34.257688 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.250462    2570 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 29 03:55:34.257688 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.252274    2570 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 29 03:55:34.257688 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.253574    2570 server.go:1019] "Starting client certificate rotation"
Jul 29 03:55:34.257688 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.253676    2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 29 03:55:34.257688 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.253741    2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 29 03:55:34.258034 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.255989    2570 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 29 03:55:34.258034 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.257059    2570 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 29 03:55:34.264456 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.264441    2570 log.go:25] "Validated CRI v1 runtime API"
Jul 29 03:55:34.269793 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.269778    2570 log.go:25] "Validated CRI v1 image API"
Jul 29 03:55:34.270567 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.270544    2570 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 29 03:55:34.272989 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.272972    2570 fs.go:135] Filesystem UUIDs: map[5962a902-b56c-49e7-9e71-8a74c3756c6c:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 c13ca86a-e462-44aa-a909-678ea82fa435:/dev/nvme0n1p3]
Jul 29 03:55:34.273048 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.272988    2570 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 29 03:55:34.279105 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.278988    2570 manager.go:216] Machine: {Timestamp:2026-07-29 03:55:34.277576983 +0000 UTC m=+0.332719090 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3114662 MemoryCapacity:32812171264 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec237f752ab46169a8d75d7b680cc5d9 SystemUUID:ec237f75-2ab4-6169-a8d7-5d7b680cc5d9 BootID:7429b207-2d56-4cea-a3d7-eb64ee11e93c Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:48:d1:7e:9a:0d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:48:d1:7e:9a:0d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:86:81:d4:87:ce:ca Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812171264 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 29 03:55:34.279105 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.279097    2570 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 29 03:55:34.279261 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.279202    2570 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 29 03:55:34.280006 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.279987    2570 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 29 03:55:34.282368 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.282343    2570 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 29 03:55:34.282496 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.282371    2570 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-138.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 29 03:55:34.283262 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.283252    2570 topology_manager.go:138] "Creating topology manager with none policy"
Jul 29 03:55:34.283296 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.283265    2570 container_manager_linux.go:309] "Creating device plugin manager"
Jul 29 03:55:34.283296 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.283272    2570 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 29 03:55:34.284262 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.284252    2570 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 29 03:55:34.285669 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.285659    2570 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 29 03:55:34.288578 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.288566    2570 state_mem.go:36] "Initialized new in-memory state store"
Jul 29 03:55:34.288706 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.288696    2570 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 29 03:55:34.288744 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.288727    2570 kubelet.go:486] "Attempting to sync node with API server"
Jul 29 03:55:34.288744 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.288738    2570 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 29 03:55:34.288819 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.288756    2570 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 29 03:55:34.288819 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.288766    2570 kubelet.go:398] "Adding apiserver pod source"
Jul 29 03:55:34.288819 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.288775    2570 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 29 03:55:34.289161 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.289148    2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 29 03:55:34.289210 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.289167    2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 29 03:55:34.291457 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.291427    2570 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 29 03:55:34.292730 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.292717    2570 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 29 03:55:34.292792 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.292746    2570 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 29 03:55:34.293400 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.293389    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 29 03:55:34.293435 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.293406    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 29 03:55:34.293435 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.293412    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 29 03:55:34.293435 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.293417    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 29 03:55:34.293435 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.293426    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 29 03:55:34.293435 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.293432    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 29 03:55:34.293569 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.293554    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 29 03:55:34.293569 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.293559    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 29 03:55:34.293569 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.293566    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 29 03:55:34.293683 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.293571    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 29 03:55:34.293683 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.293584    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 29 03:55:34.293683 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.293592    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 29 03:55:34.294415 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.294406    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 29 03:55:34.294415 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.294415    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 29 03:55:34.297581 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.297565    2570 server.go:1268] "Started kubelet"
Jul 29 03:55:34.297671 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.297649    2570 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 29 03:55:34.297786 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.297740    2570 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 29 03:55:34.297861 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.297833    2570 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 29 03:55:34.298491 ip-10-0-135-138 systemd[1]: Started Kubernetes Kubelet.
Jul 29 03:55:34.299596 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.299556    2570 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 29 03:55:34.299864 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.299846    2570 server.go:310] "Adding debug handlers to kubelet server"
Jul 29 03:55:34.300779 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.300754    2570 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-138.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 29 03:55:34.301250 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:34.301231    2570 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-138.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 29 03:55:34.301322 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:34.301232    2570 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 29 03:55:34.302621 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.302602    2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jdc7p"
Jul 29 03:55:34.303829 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:34.303790    2570 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 29 03:55:34.304592 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:34.303632    2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-138.ec2.internal.18c6a6777541abbf  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-138.ec2.internal,UID:ip-10-0-135-138.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-138.ec2.internal,},FirstTimestamp:2026-07-29 03:55:34.297537471 +0000 UTC m=+0.352679577,LastTimestamp:2026-07-29 03:55:34.297537471 +0000 UTC m=+0.352679577,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-138.ec2.internal,}"
Jul 29 03:55:34.306371 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.306353    2570 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 29 03:55:34.306371 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.306363    2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 29 03:55:34.306778 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:34.306586    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-138.ec2.internal\" not found"
Jul 29 03:55:34.306981 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.306964    2570 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 29 03:55:34.307051 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.306984    2570 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 29 03:55:34.307117 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.307083    2570 reconstruct.go:97] "Volume reconstruction finished"
Jul 29 03:55:34.307117 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.307090    2570 reconciler.go:29] "Reconciler: start to sync state"
Jul 29 03:55:34.307245 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.307176    2570 factory.go:55] Registering systemd factory
Jul 29 03:55:34.307245 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.307190    2570 factory.go:223] Registration of the systemd container factory successfully
Jul 29 03:55:34.307655 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.307641    2570 factory.go:153] Registering CRI-O factory
Jul 29 03:55:34.307655 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.307657    2570 factory.go:223] Registration of the crio container factory successfully
Jul 29 03:55:34.307778 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.307724    2570 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 29 03:55:34.307778 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.307746    2570 factory.go:103] Registering Raw factory
Jul 29 03:55:34.307778 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.307762    2570 manager.go:1196] Started watching for new ooms in manager
Jul 29 03:55:34.308319 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.308304    2570 manager.go:349] Starting recovery of all containers
Jul 29 03:55:34.308452 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.308435    2570 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 29 03:55:34.311321 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:34.311293    2570 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-138.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 29 03:55:34.311418 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:34.311404    2570 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 29 03:55:34.312116 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.312095    2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jdc7p"
Jul 29 03:55:34.318198 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.318171    2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 29 03:55:34.321626 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.321492    2570 manager.go:354] Recovery completed
Jul 29 03:55:34.326069 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.326054    2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 03:55:34.327791 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.327774    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-138.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 03:55:34.327897 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.327828    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-138.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 03:55:34.327897 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.327844    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-138.ec2.internal" event="NodeHasSufficientPID"
Jul 29 03:55:34.328349 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.328335    2570 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 29 03:55:34.328349 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.328346    2570 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 29 03:55:34.328349 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.328346    2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 03:55:34.328516 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.328359    2570 state_mem.go:36] "Initialized new in-memory state store"
Jul 29 03:55:34.329027 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.329013    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-138.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 03:55:34.329093 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.329042    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-138.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 03:55:34.329093 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.329056    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-138.ec2.internal" event="NodeHasSufficientPID"
Jul 29 03:55:34.329600 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.329587    2570 policy_none.go:49] "None policy: Start"
Jul 29 03:55:34.329654 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.329604    2570 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 29 03:55:34.329654 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.329623    2570 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 29 03:55:34.329856 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:34.329735    2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-138.ec2.internal.18c6a677770f55b4  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-138.ec2.internal,UID:ip-10-0-135-138.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-138.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-138.ec2.internal,},FirstTimestamp:2026-07-29 03:55:34.327793076 +0000 UTC m=+0.382935188,LastTimestamp:2026-07-29 03:55:34.327793076 +0000 UTC m=+0.382935188,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-138.ec2.internal,}"
Jul 29 03:55:34.330842 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.330831    2570 policy_none.go:47] "Start"
Jul 29 03:55:34.366328 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.366313    2570 manager.go:337] "Starting Device Plugin manager"
Jul 29 03:55:34.366399 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:34.366341    2570 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 29 03:55:34.366399 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.366351    2570 server.go:87] "Starting device plugin registration server"
Jul 29 03:55:34.366557 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.366535    2570 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 29 03:55:34.366637 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.366557    2570 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 29 03:55:34.366692 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.366644    2570 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 29 03:55:34.366743 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.366731    2570 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 29 03:55:34.366743 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.366742    2570 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 29 03:55:34.367217 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:34.367196    2570 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 29 03:55:34.367288 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:34.367252    2570 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-138.ec2.internal\" not found"
Jul 29 03:55:34.467374 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.467348    2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 03:55:34.468711 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.468691    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-138.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 03:55:34.468791 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.468725    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-138.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 03:55:34.468791 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.468738    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-138.ec2.internal" event="NodeHasSufficientPID"
Jul 29 03:55:34.468791 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.468773    2570 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-138.ec2.internal"
Jul 29 03:55:34.476144 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.476131    2570 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-138.ec2.internal"
Jul 29 03:55:34.476188 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:34.476150    2570 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-138.ec2.internal\": node \"ip-10-0-135-138.ec2.internal\" not found"
Jul 29 03:55:34.491998 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:34.491956    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-138.ec2.internal\" not found"
Jul 29 03:55:34.503232 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.503210    2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 29 03:55:34.503316 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.503249    2570 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 29 03:55:34.503316 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.503268    2570 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 29 03:55:34.503316 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:34.503296    2570 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 29 03:55:34.504919 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.504902    2570 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 03:55:34.592311 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:34.592288    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-138.ec2.internal\" not found"
Jul 29 03:55:34.603413 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.603368    2570 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-138.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-138.ec2.internal"]
Jul 29 03:55:34.603512 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.603458    2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 03:55:34.604321 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.604306    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-138.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 03:55:34.604377 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.604352    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-138.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 03:55:34.604377 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.604369    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-138.ec2.internal" event="NodeHasSufficientPID"
Jul 29 03:55:34.605532 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.605520    2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 03:55:34.605696 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.605684    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-138.ec2.internal"
Jul 29 03:55:34.605752 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.605743    2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 03:55:34.606331 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.606313    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-138.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 03:55:34.606424 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.606358    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-138.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 03:55:34.606424 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.606379    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-138.ec2.internal" event="NodeHasSufficientPID"
Jul 29 03:55:34.606424 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.606389    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-138.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 03:55:34.606424 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.606412    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-138.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 03:55:34.606424 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.606425    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-138.ec2.internal" event="NodeHasSufficientPID"
Jul 29 03:55:34.607598 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.607582    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-138.ec2.internal"
Jul 29 03:55:34.607686 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.607608    2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 03:55:34.608256 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.608242    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-138.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 03:55:34.608333 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.608261    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-138.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 03:55:34.608333 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.608272    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-138.ec2.internal" event="NodeHasSufficientPID"
Jul 29 03:55:34.636475 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:34.636461    2570 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-138.ec2.internal\" not found" node="ip-10-0-135-138.ec2.internal"
Jul 29 03:55:34.639870 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:34.639855    2570 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-138.ec2.internal\" not found" node="ip-10-0-135-138.ec2.internal"
Jul 29 03:55:34.692543 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:34.692528    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-138.ec2.internal\" not found"
Jul 29 03:55:34.708862 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.708843    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f49f38cf04909602874266f431d33228-config\") pod \"kube-apiserver-proxy-ip-10-0-135-138.ec2.internal\" (UID: \"f49f38cf04909602874266f431d33228\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-138.ec2.internal"
Jul 29 03:55:34.708920 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.708868    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3fb5aae1a0930c70f1e97b9429e03a80-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-138.ec2.internal\" (UID: \"3fb5aae1a0930c70f1e97b9429e03a80\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-138.ec2.internal"
Jul 29 03:55:34.708920 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.708885    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3fb5aae1a0930c70f1e97b9429e03a80-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-138.ec2.internal\" (UID: \"3fb5aae1a0930c70f1e97b9429e03a80\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-138.ec2.internal"
Jul 29 03:55:34.793263 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:34.793209    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-138.ec2.internal\" not found"
Jul 29 03:55:34.809526 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.809501    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f49f38cf04909602874266f431d33228-config\") pod \"kube-apiserver-proxy-ip-10-0-135-138.ec2.internal\" (UID: \"f49f38cf04909602874266f431d33228\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-138.ec2.internal"
Jul 29 03:55:34.809590 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.809530    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3fb5aae1a0930c70f1e97b9429e03a80-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-138.ec2.internal\" (UID: \"3fb5aae1a0930c70f1e97b9429e03a80\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-138.ec2.internal"
Jul 29 03:55:34.809590 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.809544    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3fb5aae1a0930c70f1e97b9429e03a80-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-138.ec2.internal\" (UID: \"3fb5aae1a0930c70f1e97b9429e03a80\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-138.ec2.internal"
Jul 29 03:55:34.809685 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.809589    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3fb5aae1a0930c70f1e97b9429e03a80-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-138.ec2.internal\" (UID: \"3fb5aae1a0930c70f1e97b9429e03a80\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-138.ec2.internal"
Jul 29 03:55:34.809685 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.809633    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f49f38cf04909602874266f431d33228-config\") pod \"kube-apiserver-proxy-ip-10-0-135-138.ec2.internal\" (UID: \"f49f38cf04909602874266f431d33228\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-138.ec2.internal"
Jul 29 03:55:34.809685 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.809630    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3fb5aae1a0930c70f1e97b9429e03a80-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-138.ec2.internal\" (UID: \"3fb5aae1a0930c70f1e97b9429e03a80\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-138.ec2.internal"
Jul 29 03:55:34.893700 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:34.893668    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-138.ec2.internal\" not found"
Jul 29 03:55:34.939236 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.939199    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-138.ec2.internal"
Jul 29 03:55:34.943567 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:34.943451    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-138.ec2.internal"
Jul 29 03:55:34.994619 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:34.994592    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-138.ec2.internal\" not found"
Jul 29 03:55:35.095206 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:35.095156    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-138.ec2.internal\" not found"
Jul 29 03:55:35.117378 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:35.117357    2570 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 03:55:35.123035 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:35.123021    2570 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 03:55:35.207291 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:35.207274    2570 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-138.ec2.internal"
Jul 29 03:55:35.217534 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:35.217514    2570 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 29 03:55:35.219543 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:35.219531    2570 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-138.ec2.internal"
Jul 29 03:55:35.224860 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:35.224843    2570 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 29 03:55:35.253004 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:35.252976    2570 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 29 03:55:35.253368 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:35.253094    2570 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 29 03:55:35.253368 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:35.253137    2570 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 29 03:55:35.253368 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:35.253146    2570 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 29 03:55:35.289304 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:35.289281    2570 apiserver.go:52] "Watching apiserver"
Jul 29 03:55:35.307220 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:35.307198    2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 29 03:55:35.309843 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:35.309824    2570 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 29 03:55:35.310934 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:35.310914    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-k5hmf","kube-system/kube-apiserver-proxy-ip-10-0-135-138.ec2.internal","openshift-cluster-node-tuning-operator/tuned-8p4vn","openshift-image-registry/node-ca-kkpf7","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-138.ec2.internal","openshift-multus/multus-additional-cni-plugins-5ddp8","openshift-multus/network-metrics-daemon-5ddj8","openshift-network-diagnostics/network-check-target-8222s","kube-system/global-pull-secret-syncer-vvzjp","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8mn5m","openshift-dns/node-resolver-hc29h","openshift-multus/multus-dc668","openshift-network-operator/iptables-alerter-66gml","openshift-ovn-kubernetes/ovnkube-node-pznfg"]
Jul 29 03:55:35.312633 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:35.312619    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ddj8"
Jul 29 03:55:35.312698 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:35.312681    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ddj8" podUID="64f62499-e7b7-4249-a1e4-636144783b55"
Jul 29 03:55:35.313734 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:35.313719    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hc29h"
Jul 29 03:55:35.314862 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:35.314842    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8p4vn"
Jul 29 03:55:35.316049 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:35.316033    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kkpf7"
Jul 29 03:55:35.316241 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:35.316221    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 29 03:55:35.316536 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:35.316517    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 29 03:55:35.316536 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:35.316528    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-25jxj\""
Jul 29 03:55:35.317443 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:35.317414    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 29 03:55:35.317541 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:35.317411    2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-28 03:50:35 +0000 UTC" deadline="2028-03-09 15:45:49.356091415 +0000 UTC"
Jul 29 03:55:35.317541 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:35.317448    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 29 03:55:35.317541 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:35.317457    2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14147h50m14.038637943s"
Jul 29 03:55:35.317541 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:35.317442    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bgncs\""
Jul 29 03:55:35.317701 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:35.317687    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pznfg"
Jul 29 03:55:35.318484 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:35.318469    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 29 03:55:35.318771 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:35.318756    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 29 03:55:35.318846 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:35.318782    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 29 03:55:35.318901 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:35.318880    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lm8lv\""
Jul 29 03:55:35.318901 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:35.318890    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5ddp8"
Jul 29 03:55:35.320336 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:35.320320    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vvzjp"
Jul 29 03:55:35.320422 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:35.320362    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vvzjp" podUID="d9e67808-ec21-42a2-bbaf-a28164326ccc"
Jul 29 03:55:35.321002 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:35.320982    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 29 03:55:35.321092 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:35.321056    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-zbh6s\""
Jul 29 03:55:35.321214 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:35.321203    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 29 03:55:35.321456 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:35.321445    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-66gml"
Jul 29 03:55:35.321981 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:35.321963    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 29 03:55:35.321981 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:35.321975    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 29 03:55:35.322096 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:35.321967    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 29 03:55:35.322096 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:35.322019    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 29 03:55:35.322096 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:35.322038    2570 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 29 03:55:35.322290 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:35.322276    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 29 03:55:35.322290 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:35.322287    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 29 03:55:35.322388 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:35.322310    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 29 03:55:35.322388 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:35.322280    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-pv7zv\""
Jul 29 03:55:35.322388 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:35.322369    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 29 03:55:35.322579 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:35.322565    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 29 03:55:35.322834 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:35.322820    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8222s"
Jul 29 03:55:35.322933 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:35.322917    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8222s" podUID="a0ccb95e-5e5c-4a10-9f35-6510b59e84aa"
Jul 29 03:55:35.323685 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:35.323673    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 29 03:55:35.324073 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:35.324060    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 29 03:55:35.324165 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:35.324150    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8mn5m"
Jul 29 03:55:35.324246 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:35.324234    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-kffgr\""
Jul 29 03:55:35.324576 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:35.324563    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 29 03:55:35.325452 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:35.325439    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-k5hmf"
Jul 29 03:55:35.327831 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:35.326905    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 29 03:55:35.327831 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:35.327056    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dc668"
Jul 29 03:55:35.327831 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:35.327083    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 29 03:55:35.327831 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:35.327206    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-wfc7m\""
Jul 29 03:55:35.327831 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:35.327457    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 29 03:55:35.328430 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:35.328412    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 29 03:55:35.328575 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:35.328554    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-zwbqx\""
Jul 29 03:55:35.328961 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:35.328946    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 29 03:55:35.329833 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:35.329800    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 29 03:55:35.330255 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:35.330241    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-x7ncc\""
Jul 29 03:55:36.671650 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.671634    2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-fctmg"
Jul 29 03:55:36.682094 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.682079    2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-fctmg"
Jul 29 03:55:36.734179 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.734164    2570 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 29 03:55:36.737697 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.737680    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a5135b47-d2a7-45d5-ab3b-cb4404d1c00b-cni-binary-copy\") pod \"multus-additional-cni-plugins-5ddp8\" (UID: \"a5135b47-d2a7-45d5-ab3b-cb4404d1c00b\") " pod="openshift-multus/multus-additional-cni-plugins-5ddp8"
Jul 29 03:55:36.737773 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.737716    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/179b4cec-34c3-4de7-9ff6-c9faee097c87-multus-daemon-config\") pod \"multus-dc668\" (UID: \"179b4cec-34c3-4de7-9ff6-c9faee097c87\") " pod="openshift-multus/multus-dc668"
Jul 29 03:55:36.737773 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.737741    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/179b4cec-34c3-4de7-9ff6-c9faee097c87-host-run-multus-certs\") pod \"multus-dc668\" (UID: \"179b4cec-34c3-4de7-9ff6-c9faee097c87\") " pod="openshift-multus/multus-dc668"
Jul 29 03:55:36.737773 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.737766    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/34dcf361-8ea0-4b35-a259-0d0ab4ba5052-etc-sysctl-conf\") pod \"tuned-8p4vn\" (UID: \"34dcf361-8ea0-4b35-a259-0d0ab4ba5052\") " pod="openshift-cluster-node-tuning-operator/tuned-8p4vn"
Jul 29 03:55:36.737943 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.737786    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/34dcf361-8ea0-4b35-a259-0d0ab4ba5052-run\") pod \"tuned-8p4vn\" (UID: \"34dcf361-8ea0-4b35-a259-0d0ab4ba5052\") " pod="openshift-cluster-node-tuning-operator/tuned-8p4vn"
Jul 29 03:55:36.737943 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.737807    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/34dcf361-8ea0-4b35-a259-0d0ab4ba5052-sys\") pod \"tuned-8p4vn\" (UID: \"34dcf361-8ea0-4b35-a259-0d0ab4ba5052\") " pod="openshift-cluster-node-tuning-operator/tuned-8p4vn"
Jul 29 03:55:36.737943 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.737848    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/34dcf361-8ea0-4b35-a259-0d0ab4ba5052-var-lib-kubelet\") pod \"tuned-8p4vn\" (UID: \"34dcf361-8ea0-4b35-a259-0d0ab4ba5052\") " pod="openshift-cluster-node-tuning-operator/tuned-8p4vn"
Jul 29 03:55:36.737943 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.737871    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/11451627-c54f-414d-a818-3acd21c7c829-host-slash\") pod \"ovnkube-node-pznfg\" (UID: \"11451627-c54f-414d-a818-3acd21c7c829\") " pod="openshift-ovn-kubernetes/ovnkube-node-pznfg"
Jul 29 03:55:36.737943 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.737895    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7b3ec617-26f1-4c28-aa08-2710238467ac-serviceca\") pod \"node-ca-kkpf7\" (UID: \"7b3ec617-26f1-4c28-aa08-2710238467ac\") " pod="openshift-image-registry/node-ca-kkpf7"
Jul 29 03:55:36.737943 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.737918    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4e72f9a5-de78-4f59-b7b7-b6a1bd93c34c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8mn5m\" (UID: \"4e72f9a5-de78-4f59-b7b7-b6a1bd93c34c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8mn5m"
Jul 29 03:55:36.737943 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.737941    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4e72f9a5-de78-4f59-b7b7-b6a1bd93c34c-sys-fs\") pod \"aws-ebs-csi-driver-node-8mn5m\" (UID: \"4e72f9a5-de78-4f59-b7b7-b6a1bd93c34c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8mn5m"
Jul 29 03:55:36.738267 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.737963    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d9e67808-ec21-42a2-bbaf-a28164326ccc-dbus\") pod \"global-pull-secret-syncer-vvzjp\" (UID: \"d9e67808-ec21-42a2-bbaf-a28164326ccc\") " pod="kube-system/global-pull-secret-syncer-vvzjp"
Jul 29 03:55:36.738267 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.737986    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d7d7563a-a7bd-477c-9f04-4a4ada546abd-host-slash\") pod \"iptables-alerter-66gml\" (UID: \"d7d7563a-a7bd-477c-9f04-4a4ada546abd\") " pod="openshift-network-operator/iptables-alerter-66gml"
Jul 29 03:55:36.738267 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.738010    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7252c41b-1992-4e77-b76d-8f036a55e064-konnectivity-ca\") pod \"konnectivity-agent-k5hmf\" (UID: \"7252c41b-1992-4e77-b76d-8f036a55e064\") " pod="kube-system/konnectivity-agent-k5hmf"
Jul 29 03:55:36.738267 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.738033    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/179b4cec-34c3-4de7-9ff6-c9faee097c87-system-cni-dir\") pod \"multus-dc668\" (UID: \"179b4cec-34c3-4de7-9ff6-c9faee097c87\") " pod="openshift-multus/multus-dc668"
Jul 29 03:55:36.738267 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.738057    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmclr\" (UniqueName: \"kubernetes.io/projected/7b3ec617-26f1-4c28-aa08-2710238467ac-kube-api-access-nmclr\") pod \"node-ca-kkpf7\" (UID: \"7b3ec617-26f1-4c28-aa08-2710238467ac\") " pod="openshift-image-registry/node-ca-kkpf7"
Jul 29 03:55:36.738267 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.738080    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/179b4cec-34c3-4de7-9ff6-c9faee097c87-host-var-lib-kubelet\") pod \"multus-dc668\" (UID: \"179b4cec-34c3-4de7-9ff6-c9faee097c87\") " pod="openshift-multus/multus-dc668"
Jul 29 03:55:36.738267 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.738103    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/11451627-c54f-414d-a818-3acd21c7c829-host-run-netns\") pod \"ovnkube-node-pznfg\" (UID: \"11451627-c54f-414d-a818-3acd21c7c829\") " pod="openshift-ovn-kubernetes/ovnkube-node-pznfg"
Jul 29 03:55:36.738267 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.738126    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/11451627-c54f-414d-a818-3acd21c7c829-run-openvswitch\") pod \"ovnkube-node-pznfg\" (UID: \"11451627-c54f-414d-a818-3acd21c7c829\") " pod="openshift-ovn-kubernetes/ovnkube-node-pznfg"
Jul 29 03:55:36.738267 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.738160    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/179b4cec-34c3-4de7-9ff6-c9faee097c87-hostroot\") pod \"multus-dc668\" (UID: \"179b4cec-34c3-4de7-9ff6-c9faee097c87\") " pod="openshift-multus/multus-dc668"
Jul 29 03:55:36.738267 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.738181    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wt4rs\" (UniqueName: \"kubernetes.io/projected/64f62499-e7b7-4249-a1e4-636144783b55-kube-api-access-wt4rs\") pod \"network-metrics-daemon-5ddj8\" (UID: \"64f62499-e7b7-4249-a1e4-636144783b55\") " pod="openshift-multus/network-metrics-daemon-5ddj8"
Jul 29 03:55:36.738267 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.738203    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/34dcf361-8ea0-4b35-a259-0d0ab4ba5052-host\") pod \"tuned-8p4vn\" (UID: \"34dcf361-8ea0-4b35-a259-0d0ab4ba5052\") " pod="openshift-cluster-node-tuning-operator/tuned-8p4vn"
Jul 29 03:55:36.738267 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.738230    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/11451627-c54f-414d-a818-3acd21c7c829-host-kubelet\") pod \"ovnkube-node-pznfg\" (UID: \"11451627-c54f-414d-a818-3acd21c7c829\") " pod="openshift-ovn-kubernetes/ovnkube-node-pznfg"
Jul 29 03:55:36.738267 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.738256    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d9e67808-ec21-42a2-bbaf-a28164326ccc-kubelet-config\") pod \"global-pull-secret-syncer-vvzjp\" (UID: \"d9e67808-ec21-42a2-bbaf-a28164326ccc\") " pod="kube-system/global-pull-secret-syncer-vvzjp"
Jul 29 03:55:36.738732 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.738278    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfgm9\" (UniqueName: \"kubernetes.io/projected/d7d7563a-a7bd-477c-9f04-4a4ada546abd-kube-api-access-hfgm9\") pod \"iptables-alerter-66gml\" (UID: \"d7d7563a-a7bd-477c-9f04-4a4ada546abd\") " pod="openshift-network-operator/iptables-alerter-66gml"
Jul 29 03:55:36.738732 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.738309    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a5135b47-d2a7-45d5-ab3b-cb4404d1c00b-system-cni-dir\") pod \"multus-additional-cni-plugins-5ddp8\" (UID: \"a5135b47-d2a7-45d5-ab3b-cb4404d1c00b\") " pod="openshift-multus/multus-additional-cni-plugins-5ddp8"
Jul 29 03:55:36.738732 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.738331    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4e72f9a5-de78-4f59-b7b7-b6a1bd93c34c-socket-dir\") pod \"aws-ebs-csi-driver-node-8mn5m\" (UID: \"4e72f9a5-de78-4f59-b7b7-b6a1bd93c34c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8mn5m"
Jul 29 03:55:36.738732 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.738369    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/11451627-c54f-414d-a818-3acd21c7c829-host-cni-bin\") pod \"ovnkube-node-pznfg\" (UID: \"11451627-c54f-414d-a818-3acd21c7c829\") " pod="openshift-ovn-kubernetes/ovnkube-node-pznfg"
Jul 29 03:55:36.738732 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.738400    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/34dcf361-8ea0-4b35-a259-0d0ab4ba5052-lib-modules\") pod \"tuned-8p4vn\" (UID: \"34dcf361-8ea0-4b35-a259-0d0ab4ba5052\") " pod="openshift-cluster-node-tuning-operator/tuned-8p4vn"
Jul 29 03:55:36.738732 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.738422    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/34dcf361-8ea0-4b35-a259-0d0ab4ba5052-tmp\") pod \"tuned-8p4vn\" (UID: \"34dcf361-8ea0-4b35-a259-0d0ab4ba5052\") " pod="openshift-cluster-node-tuning-operator/tuned-8p4vn"
Jul 29 03:55:36.738732 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.738451    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9nwl\" (UniqueName: \"kubernetes.io/projected/34dcf361-8ea0-4b35-a259-0d0ab4ba5052-kube-api-access-b9nwl\") pod \"tuned-8p4vn\" (UID: \"34dcf361-8ea0-4b35-a259-0d0ab4ba5052\") " pod="openshift-cluster-node-tuning-operator/tuned-8p4vn"
Jul 29 03:55:36.738732 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.738475    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/34dcf361-8ea0-4b35-a259-0d0ab4ba5052-etc-sysctl-d\") pod \"tuned-8p4vn\" (UID: \"34dcf361-8ea0-4b35-a259-0d0ab4ba5052\") " pod="openshift-cluster-node-tuning-operator/tuned-8p4vn"
Jul 29 03:55:36.738732 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.738496    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a5135b47-d2a7-45d5-ab3b-cb4404d1c00b-cnibin\") pod \"multus-additional-cni-plugins-5ddp8\" (UID: \"a5135b47-d2a7-45d5-ab3b-cb4404d1c00b\") " pod="openshift-multus/multus-additional-cni-plugins-5ddp8"
Jul 29 03:55:36.738732 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.738520    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/179b4cec-34c3-4de7-9ff6-c9faee097c87-os-release\") pod \"multus-dc668\" (UID: \"179b4cec-34c3-4de7-9ff6-c9faee097c87\") " pod="openshift-multus/multus-dc668"
Jul 29 03:55:36.738732 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.738541    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/179b4cec-34c3-4de7-9ff6-c9faee097c87-host-run-k8s-cni-cncf-io\") pod \"multus-dc668\" (UID: \"179b4cec-34c3-4de7-9ff6-c9faee097c87\") " pod="openshift-multus/multus-dc668"
Jul 29 03:55:36.738732 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.738561    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/179b4cec-34c3-4de7-9ff6-c9faee097c87-host-run-netns\") pod \"multus-dc668\" (UID: \"179b4cec-34c3-4de7-9ff6-c9faee097c87\") " pod="openshift-multus/multus-dc668"
Jul 29 03:55:36.738732 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.738574    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/179b4cec-34c3-4de7-9ff6-c9faee097c87-multus-conf-dir\") pod \"multus-dc668\" (UID: \"179b4cec-34c3-4de7-9ff6-c9faee097c87\") " pod="openshift-multus/multus-dc668"
Jul 29 03:55:36.738732 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.738593    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/11451627-c54f-414d-a818-3acd21c7c829-etc-openvswitch\") pod \"ovnkube-node-pznfg\" (UID: \"11451627-c54f-414d-a818-3acd21c7c829\") " pod="openshift-ovn-kubernetes/ovnkube-node-pznfg"
Jul 29 03:55:36.738732 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.738614    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a5135b47-d2a7-45d5-ab3b-cb4404d1c00b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5ddp8\" (UID: \"a5135b47-d2a7-45d5-ab3b-cb4404d1c00b\") " pod="openshift-multus/multus-additional-cni-plugins-5ddp8"
Jul 29 03:55:36.738732 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.738627    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77l56\" (UniqueName: \"kubernetes.io/projected/179b4cec-34c3-4de7-9ff6-c9faee097c87-kube-api-access-77l56\") pod \"multus-dc668\" (UID: \"179b4cec-34c3-4de7-9ff6-c9faee097c87\") " pod="openshift-multus/multus-dc668"
Jul 29 03:55:36.739271 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.738640    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/11451627-c54f-414d-a818-3acd21c7c829-node-log\") pod \"ovnkube-node-pznfg\" (UID: \"11451627-c54f-414d-a818-3acd21c7c829\") " pod="openshift-ovn-kubernetes/ovnkube-node-pznfg"
Jul 29 03:55:36.739271 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.738659    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/11451627-c54f-414d-a818-3acd21c7c829-env-overrides\") pod \"ovnkube-node-pznfg\" (UID: \"11451627-c54f-414d-a818-3acd21c7c829\") " pod="openshift-ovn-kubernetes/ovnkube-node-pznfg"
Jul 29 03:55:36.739271 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.738688    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/11451627-c54f-414d-a818-3acd21c7c829-ovnkube-script-lib\") pod \"ovnkube-node-pznfg\" (UID: \"11451627-c54f-414d-a818-3acd21c7c829\") " pod="openshift-ovn-kubernetes/ovnkube-node-pznfg"
Jul 29 03:55:36.739271 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.738725    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/11451627-c54f-414d-a818-3acd21c7c829-ovn-node-metrics-cert\") pod \"ovnkube-node-pznfg\" (UID: \"11451627-c54f-414d-a818-3acd21c7c829\") " pod="openshift-ovn-kubernetes/ovnkube-node-pznfg"
Jul 29 03:55:36.739271 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.738771    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7252c41b-1992-4e77-b76d-8f036a55e064-agent-certs\") pod \"konnectivity-agent-k5hmf\" (UID: \"7252c41b-1992-4e77-b76d-8f036a55e064\") " pod="kube-system/konnectivity-agent-k5hmf"
Jul 29 03:55:36.739271 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.738788    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbdtr\" (UniqueName: \"kubernetes.io/projected/4e72f9a5-de78-4f59-b7b7-b6a1bd93c34c-kube-api-access-xbdtr\") pod \"aws-ebs-csi-driver-node-8mn5m\" (UID: \"4e72f9a5-de78-4f59-b7b7-b6a1bd93c34c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8mn5m"
Jul 29 03:55:36.739271 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.738811    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/179b4cec-34c3-4de7-9ff6-c9faee097c87-host-var-lib-cni-bin\") pod \"multus-dc668\" (UID: \"179b4cec-34c3-4de7-9ff6-c9faee097c87\") " pod="openshift-multus/multus-dc668"
Jul 29 03:55:36.739271 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.738847    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/11451627-c54f-414d-a818-3acd21c7c829-systemd-units\") pod \"ovnkube-node-pznfg\" (UID: \"11451627-c54f-414d-a818-3acd21c7c829\") " pod="openshift-ovn-kubernetes/ovnkube-node-pznfg"
Jul 29 03:55:36.739271 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.738864    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/11451627-c54f-414d-a818-3acd21c7c829-log-socket\") pod \"ovnkube-node-pznfg\" (UID: \"11451627-c54f-414d-a818-3acd21c7c829\") " pod="openshift-ovn-kubernetes/ovnkube-node-pznfg"
Jul 29 03:55:36.739271 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.738877    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11451627-c54f-414d-a818-3acd21c7c829-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pznfg\" (UID: \"11451627-c54f-414d-a818-3acd21c7c829\") " pod="openshift-ovn-kubernetes/ovnkube-node-pznfg"
Jul 29 03:55:36.739271 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.738891    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4e72f9a5-de78-4f59-b7b7-b6a1bd93c34c-etc-selinux\") pod \"aws-ebs-csi-driver-node-8mn5m\" (UID: \"4e72f9a5-de78-4f59-b7b7-b6a1bd93c34c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8mn5m"
Jul 29 03:55:36.739271 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.738905    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a5135b47-d2a7-45d5-ab3b-cb4404d1c00b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5ddp8\" (UID: \"a5135b47-d2a7-45d5-ab3b-cb4404d1c00b\") " pod="openshift-multus/multus-additional-cni-plugins-5ddp8"
Jul 29 03:55:36.739271 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.738923    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c00b9ff9-d986-4043-8403-c8d9a68596da-hosts-file\") pod \"node-resolver-hc29h\" (UID: \"c00b9ff9-d986-4043-8403-c8d9a68596da\") " pod="openshift-dns/node-resolver-hc29h"
Jul 29 03:55:36.739271 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.738940    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c00b9ff9-d986-4043-8403-c8d9a68596da-tmp-dir\") pod \"node-resolver-hc29h\" (UID: \"c00b9ff9-d986-4043-8403-c8d9a68596da\") " pod="openshift-dns/node-resolver-hc29h"
Jul 29 03:55:36.739271 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.738958    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/179b4cec-34c3-4de7-9ff6-c9faee097c87-host-var-lib-cni-multus\") pod \"multus-dc668\" (UID: \"179b4cec-34c3-4de7-9ff6-c9faee097c87\") " pod="openshift-multus/multus-dc668"
Jul 29 03:55:36.739271 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.738973    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/179b4cec-34c3-4de7-9ff6-c9faee097c87-etc-kubernetes\") pod \"multus-dc668\" (UID: \"179b4cec-34c3-4de7-9ff6-c9faee097c87\") " pod="openshift-multus/multus-dc668"
Jul 29 03:55:36.739740 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.738986    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4e72f9a5-de78-4f59-b7b7-b6a1bd93c34c-device-dir\") pod \"aws-ebs-csi-driver-node-8mn5m\" (UID: \"4e72f9a5-de78-4f59-b7b7-b6a1bd93c34c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8mn5m"
Jul 29 03:55:36.739740 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.739026    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/34dcf361-8ea0-4b35-a259-0d0ab4ba5052-etc-sysconfig\") pod \"tuned-8p4vn\" (UID: \"34dcf361-8ea0-4b35-a259-0d0ab4ba5052\") " pod="openshift-cluster-node-tuning-operator/tuned-8p4vn"
Jul 29 03:55:36.739740 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.739056    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/34dcf361-8ea0-4b35-a259-0d0ab4ba5052-etc-kubernetes\") pod \"tuned-8p4vn\" (UID: \"34dcf361-8ea0-4b35-a259-0d0ab4ba5052\") " pod="openshift-cluster-node-tuning-operator/tuned-8p4vn"
Jul 29 03:55:36.739740 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.739075    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/11451627-c54f-414d-a818-3acd21c7c829-var-lib-openvswitch\") pod \"ovnkube-node-pznfg\" (UID: \"11451627-c54f-414d-a818-3acd21c7c829\") " pod="openshift-ovn-kubernetes/ovnkube-node-pznfg"
Jul 29 03:55:36.739740 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.739091    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/11451627-c54f-414d-a818-3acd21c7c829-host-cni-netd\") pod \"ovnkube-node-pznfg\" (UID: \"11451627-c54f-414d-a818-3acd21c7c829\") " pod="openshift-ovn-kubernetes/ovnkube-node-pznfg"
Jul 29 03:55:36.739740 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.739111    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d7d7563a-a7bd-477c-9f04-4a4ada546abd-iptables-alerter-script\") pod \"iptables-alerter-66gml\" (UID: \"d7d7563a-a7bd-477c-9f04-4a4ada546abd\") " pod="openshift-network-operator/iptables-alerter-66gml"
Jul 29 03:55:36.739740 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.739125    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a5135b47-d2a7-45d5-ab3b-cb4404d1c00b-os-release\") pod \"multus-additional-cni-plugins-5ddp8\" (UID: \"a5135b47-d2a7-45d5-ab3b-cb4404d1c00b\") " pod="openshift-multus/multus-additional-cni-plugins-5ddp8"
Jul 29 03:55:36.739740 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.739147    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a5135b47-d2a7-45d5-ab3b-cb4404d1c00b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5ddp8\" (UID: \"a5135b47-d2a7-45d5-ab3b-cb4404d1c00b\") " pod="openshift-multus/multus-additional-cni-plugins-5ddp8"
Jul 29 03:55:36.739740 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.739169    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/179b4cec-34c3-4de7-9ff6-c9faee097c87-multus-cni-dir\") pod \"multus-dc668\" (UID: \"179b4cec-34c3-4de7-9ff6-c9faee097c87\") " pod="openshift-multus/multus-dc668"
Jul 29 03:55:36.739740 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.739182    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/179b4cec-34c3-4de7-9ff6-c9faee097c87-cnibin\") pod \"multus-dc668\" (UID: \"179b4cec-34c3-4de7-9ff6-c9faee097c87\") " pod="openshift-multus/multus-dc668"
Jul 29 03:55:36.739740 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.739207    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/179b4cec-34c3-4de7-9ff6-c9faee097c87-cni-binary-copy\") pod \"multus-dc668\" (UID: \"179b4cec-34c3-4de7-9ff6-c9faee097c87\") " pod="openshift-multus/multus-dc668"
Jul 29 03:55:36.739740 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.739230    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64f62499-e7b7-4249-a1e4-636144783b55-metrics-certs\") pod \"network-metrics-daemon-5ddj8\" (UID: \"64f62499-e7b7-4249-a1e4-636144783b55\") " pod="openshift-multus/network-metrics-daemon-5ddj8"
Jul 29 03:55:36.739740 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.739247    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lj5r\" (UniqueName: \"kubernetes.io/projected/c00b9ff9-d986-4043-8403-c8d9a68596da-kube-api-access-4lj5r\") pod \"node-resolver-hc29h\" (UID: \"c00b9ff9-d986-4043-8403-c8d9a68596da\") " pod="openshift-dns/node-resolver-hc29h"
Jul 29 03:55:36.739740 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.739268    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcgsk\" (UniqueName: \"kubernetes.io/projected/a5135b47-d2a7-45d5-ab3b-cb4404d1c00b-kube-api-access-bcgsk\") pod \"multus-additional-cni-plugins-5ddp8\" (UID: \"a5135b47-d2a7-45d5-ab3b-cb4404d1c00b\") " pod="openshift-multus/multus-additional-cni-plugins-5ddp8"
Jul 29 03:55:36.739740 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.739282    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/11451627-c54f-414d-a818-3acd21c7c829-run-systemd\") pod \"ovnkube-node-pznfg\" (UID: \"11451627-c54f-414d-a818-3acd21c7c829\") " pod="openshift-ovn-kubernetes/ovnkube-node-pznfg"
Jul 29 03:55:36.739740 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.739297    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11451627-c54f-414d-a818-3acd21c7c829-host-run-ovn-kubernetes\") pod \"ovnkube-node-pznfg\" (UID: \"11451627-c54f-414d-a818-3acd21c7c829\") " pod="openshift-ovn-kubernetes/ovnkube-node-pznfg"
Jul 29 03:55:36.740241 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.739312    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/11451627-c54f-414d-a818-3acd21c7c829-ovnkube-config\") pod \"ovnkube-node-pznfg\" (UID: \"11451627-c54f-414d-a818-3acd21c7c829\") " pod="openshift-ovn-kubernetes/ovnkube-node-pznfg"
Jul 29 03:55:36.740241 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.739325    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/34dcf361-8ea0-4b35-a259-0d0ab4ba5052-etc-modprobe-d\") pod \"tuned-8p4vn\" (UID: \"34dcf361-8ea0-4b35-a259-0d0ab4ba5052\") " pod="openshift-cluster-node-tuning-operator/tuned-8p4vn"
Jul 29 03:55:36.740241 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.739341    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7b3ec617-26f1-4c28-aa08-2710238467ac-host\") pod \"node-ca-kkpf7\" (UID: \"7b3ec617-26f1-4c28-aa08-2710238467ac\") " pod="openshift-image-registry/node-ca-kkpf7"
Jul 29 03:55:36.740241 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.739357    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4e72f9a5-de78-4f59-b7b7-b6a1bd93c34c-registration-dir\") pod \"aws-ebs-csi-driver-node-8mn5m\" (UID: \"4e72f9a5-de78-4f59-b7b7-b6a1bd93c34c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8mn5m"
Jul 29 03:55:36.740241 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.739371    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/11451627-c54f-414d-a818-3acd21c7c829-run-ovn\") pod \"ovnkube-node-pznfg\" (UID: \"11451627-c54f-414d-a818-3acd21c7c829\") " pod="openshift-ovn-kubernetes/ovnkube-node-pznfg"
Jul 29 03:55:36.740241 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.739390    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82pqq\" (UniqueName: \"kubernetes.io/projected/11451627-c54f-414d-a818-3acd21c7c829-kube-api-access-82pqq\") pod \"ovnkube-node-pznfg\" (UID: \"11451627-c54f-414d-a818-3acd21c7c829\") " pod="openshift-ovn-kubernetes/ovnkube-node-pznfg"
Jul 29 03:55:36.740241 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.739403    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvmpk\" (UniqueName: \"kubernetes.io/projected/a0ccb95e-5e5c-4a10-9f35-6510b59e84aa-kube-api-access-hvmpk\") pod \"network-check-target-8222s\" (UID: \"a0ccb95e-5e5c-4a10-9f35-6510b59e84aa\") " pod="openshift-network-diagnostics/network-check-target-8222s"
Jul 29 03:55:36.740241 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.739423    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/179b4cec-34c3-4de7-9ff6-c9faee097c87-multus-socket-dir-parent\") pod \"multus-dc668\" (UID: \"179b4cec-34c3-4de7-9ff6-c9faee097c87\") " pod="openshift-multus/multus-dc668"
Jul 29 03:55:36.740241 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.739444    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/34dcf361-8ea0-4b35-a259-0d0ab4ba5052-etc-systemd\") pod \"tuned-8p4vn\" (UID: \"34dcf361-8ea0-4b35-a259-0d0ab4ba5052\") " pod="openshift-cluster-node-tuning-operator/tuned-8p4vn"
Jul 29 03:55:36.740241 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.739466    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/34dcf361-8ea0-4b35-a259-0d0ab4ba5052-etc-tuned\") pod \"tuned-8p4vn\" (UID: \"34dcf361-8ea0-4b35-a259-0d0ab4ba5052\") " pod="openshift-cluster-node-tuning-operator/tuned-8p4vn"
Jul 29 03:55:36.740241 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.739490    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d9e67808-ec21-42a2-bbaf-a28164326ccc-original-pull-secret\") pod \"global-pull-secret-syncer-vvzjp\" (UID: \"d9e67808-ec21-42a2-bbaf-a28164326ccc\") " pod="kube-system/global-pull-secret-syncer-vvzjp"
Jul 29 03:55:36.839885 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.839860    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/11451627-c54f-414d-a818-3acd21c7c829-ovnkube-script-lib\") pod \"ovnkube-node-pznfg\" (UID: \"11451627-c54f-414d-a818-3acd21c7c829\") " pod="openshift-ovn-kubernetes/ovnkube-node-pznfg"
Jul 29 03:55:36.839885 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.839889    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/11451627-c54f-414d-a818-3acd21c7c829-ovn-node-metrics-cert\") pod \"ovnkube-node-pznfg\" (UID: \"11451627-c54f-414d-a818-3acd21c7c829\") " pod="openshift-ovn-kubernetes/ovnkube-node-pznfg"
Jul 29 03:55:36.840129 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.839903    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7252c41b-1992-4e77-b76d-8f036a55e064-agent-certs\") pod \"konnectivity-agent-k5hmf\" (UID: \"7252c41b-1992-4e77-b76d-8f036a55e064\") " pod="kube-system/konnectivity-agent-k5hmf"
Jul 29 03:55:36.840129 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.839924    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbdtr\" (UniqueName: \"kubernetes.io/projected/4e72f9a5-de78-4f59-b7b7-b6a1bd93c34c-kube-api-access-xbdtr\") pod \"aws-ebs-csi-driver-node-8mn5m\" (UID: \"4e72f9a5-de78-4f59-b7b7-b6a1bd93c34c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8mn5m"
Jul 29 03:55:36.840129 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.839942    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/179b4cec-34c3-4de7-9ff6-c9faee097c87-host-var-lib-cni-bin\") pod \"multus-dc668\" (UID: \"179b4cec-34c3-4de7-9ff6-c9faee097c87\") " pod="openshift-multus/multus-dc668"
Jul 29 03:55:36.840129 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.839956    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/11451627-c54f-414d-a818-3acd21c7c829-systemd-units\") pod \"ovnkube-node-pznfg\" (UID: \"11451627-c54f-414d-a818-3acd21c7c829\") " pod="openshift-ovn-kubernetes/ovnkube-node-pznfg"
Jul 29 03:55:36.840129 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.839973    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/11451627-c54f-414d-a818-3acd21c7c829-log-socket\") pod \"ovnkube-node-pznfg\" (UID: \"11451627-c54f-414d-a818-3acd21c7c829\") " pod="openshift-ovn-kubernetes/ovnkube-node-pznfg"
Jul 29 03:55:36.840129 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.839997    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11451627-c54f-414d-a818-3acd21c7c829-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pznfg\" (UID: \"11451627-c54f-414d-a818-3acd21c7c829\") " pod="openshift-ovn-kubernetes/ovnkube-node-pznfg"
Jul 29 03:55:36.840129 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.840024    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4e72f9a5-de78-4f59-b7b7-b6a1bd93c34c-etc-selinux\") pod \"aws-ebs-csi-driver-node-8mn5m\" (UID: \"4e72f9a5-de78-4f59-b7b7-b6a1bd93c34c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8mn5m"
Jul 29 03:55:36.840129 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.840038    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/179b4cec-34c3-4de7-9ff6-c9faee097c87-host-var-lib-cni-bin\") pod \"multus-dc668\" (UID: \"179b4cec-34c3-4de7-9ff6-c9faee097c87\") " pod="openshift-multus/multus-dc668"
Jul 29 03:55:36.840129 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.840046    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a5135b47-d2a7-45d5-ab3b-cb4404d1c00b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5ddp8\" (UID: \"a5135b47-d2a7-45d5-ab3b-cb4404d1c00b\") " pod="openshift-multus/multus-additional-cni-plugins-5ddp8"
Jul 29 03:55:36.840129 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.840037    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/11451627-c54f-414d-a818-3acd21c7c829-systemd-units\") pod \"ovnkube-node-pznfg\" (UID: \"11451627-c54f-414d-a818-3acd21c7c829\") " pod="openshift-ovn-kubernetes/ovnkube-node-pznfg"
Jul 29 03:55:36.840129 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.840068    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c00b9ff9-d986-4043-8403-c8d9a68596da-hosts-file\") pod \"node-resolver-hc29h\" (UID: \"c00b9ff9-d986-4043-8403-c8d9a68596da\") " pod="openshift-dns/node-resolver-hc29h"
Jul 29 03:55:36.840129 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.840078    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11451627-c54f-414d-a818-3acd21c7c829-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pznfg\" (UID: \"11451627-c54f-414d-a818-3acd21c7c829\") " pod="openshift-ovn-kubernetes/ovnkube-node-pznfg"
Jul 29 03:55:36.840129 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.840093    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c00b9ff9-d986-4043-8403-c8d9a68596da-tmp-dir\") pod \"node-resolver-hc29h\" (UID: \"c00b9ff9-d986-4043-8403-c8d9a68596da\") " pod="openshift-dns/node-resolver-hc29h"
Jul 29 03:55:36.840129 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.840117    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/179b4cec-34c3-4de7-9ff6-c9faee097c87-host-var-lib-cni-multus\") pod \"multus-dc668\" (UID: \"179b4cec-34c3-4de7-9ff6-c9faee097c87\") " pod="openshift-multus/multus-dc668"
Jul 29 03:55:36.840129 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.840138    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/179b4cec-34c3-4de7-9ff6-c9faee097c87-etc-kubernetes\") pod \"multus-dc668\" (UID: \"179b4cec-34c3-4de7-9ff6-c9faee097c87\") " pod="openshift-multus/multus-dc668"
Jul 29 03:55:36.840805 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.840139    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/11451627-c54f-414d-a818-3acd21c7c829-log-socket\") pod \"ovnkube-node-pznfg\" (UID: \"11451627-c54f-414d-a818-3acd21c7c829\") " pod="openshift-ovn-kubernetes/ovnkube-node-pznfg"
Jul 29 03:55:36.840805 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.840189    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/179b4cec-34c3-4de7-9ff6-c9faee097c87-host-var-lib-cni-multus\") pod \"multus-dc668\" (UID: \"179b4cec-34c3-4de7-9ff6-c9faee097c87\") " pod="openshift-multus/multus-dc668"
Jul 29 03:55:36.840805 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.840257    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/179b4cec-34c3-4de7-9ff6-c9faee097c87-etc-kubernetes\") pod \"multus-dc668\" (UID: \"179b4cec-34c3-4de7-9ff6-c9faee097c87\") " pod="openshift-multus/multus-dc668"
Jul 29 03:55:36.840805 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.840253    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c00b9ff9-d986-4043-8403-c8d9a68596da-hosts-file\") pod \"node-resolver-hc29h\" (UID: \"c00b9ff9-d986-4043-8403-c8d9a68596da\") " pod="openshift-dns/node-resolver-hc29h"
Jul 29 03:55:36.840805 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.840269    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a5135b47-d2a7-45d5-ab3b-cb4404d1c00b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5ddp8\" (UID: \"a5135b47-d2a7-45d5-ab3b-cb4404d1c00b\") " pod="openshift-multus/multus-additional-cni-plugins-5ddp8"
Jul 29 03:55:36.840805 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.840281    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4e72f9a5-de78-4f59-b7b7-b6a1bd93c34c-device-dir\") pod \"aws-ebs-csi-driver-node-8mn5m\" (UID: \"4e72f9a5-de78-4f59-b7b7-b6a1bd93c34c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8mn5m"
Jul 29 03:55:36.840805 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.840326    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4e72f9a5-de78-4f59-b7b7-b6a1bd93c34c-device-dir\") pod \"aws-ebs-csi-driver-node-8mn5m\" (UID: \"4e72f9a5-de78-4f59-b7b7-b6a1bd93c34c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8mn5m"
Jul 29 03:55:36.840805 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.840330    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/34dcf361-8ea0-4b35-a259-0d0ab4ba5052-etc-sysconfig\") pod \"tuned-8p4vn\" (UID: \"34dcf361-8ea0-4b35-a259-0d0ab4ba5052\") " pod="openshift-cluster-node-tuning-operator/tuned-8p4vn"
Jul 29 03:55:36.840805 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.840330    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4e72f9a5-de78-4f59-b7b7-b6a1bd93c34c-etc-selinux\") pod \"aws-ebs-csi-driver-node-8mn5m\" (UID: \"4e72f9a5-de78-4f59-b7b7-b6a1bd93c34c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8mn5m"
Jul 29 03:55:36.840805 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.840354    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/34dcf361-8ea0-4b35-a259-0d0ab4ba5052-etc-kubernetes\") pod \"tuned-8p4vn\" (UID: \"34dcf361-8ea0-4b35-a259-0d0ab4ba5052\") " pod="openshift-cluster-node-tuning-operator/tuned-8p4vn"
Jul 29 03:55:36.840805 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.840375    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/11451627-c54f-414d-a818-3acd21c7c829-var-lib-openvswitch\") pod \"ovnkube-node-pznfg\" (UID: \"11451627-c54f-414d-a818-3acd21c7c829\") " pod="openshift-ovn-kubernetes/ovnkube-node-pznfg"
Jul 29 03:55:36.840805 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.840380    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/34dcf361-8ea0-4b35-a259-0d0ab4ba5052-etc-sysconfig\") pod \"tuned-8p4vn\" (UID: \"34dcf361-8ea0-4b35-a259-0d0ab4ba5052\") " pod="openshift-cluster-node-tuning-operator/tuned-8p4vn"
Jul 29 03:55:36.840805 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.840324    2570 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 29 03:55:36.840805 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.840394    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/11451627-c54f-414d-a818-3acd21c7c829-host-cni-netd\") pod \"ovnkube-node-pznfg\" (UID: \"11451627-c54f-414d-a818-3acd21c7c829\") " pod="openshift-ovn-kubernetes/ovnkube-node-pznfg"
Jul 29 03:55:36.840805 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.840430    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d7d7563a-a7bd-477c-9f04-4a4ada546abd-iptables-alerter-script\") pod \"iptables-alerter-66gml\" (UID: \"d7d7563a-a7bd-477c-9f04-4a4ada546abd\") " pod="openshift-network-operator/iptables-alerter-66gml"
Jul 29 03:55:36.840805 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.840433    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c00b9ff9-d986-4043-8403-c8d9a68596da-tmp-dir\") pod \"node-resolver-hc29h\" (UID: \"c00b9ff9-d986-4043-8403-c8d9a68596da\") " pod="openshift-dns/node-resolver-hc29h"
Jul 29 03:55:36.840805 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.840446    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a5135b47-d2a7-45d5-ab3b-cb4404d1c00b-os-release\") pod \"multus-additional-cni-plugins-5ddp8\" (UID: \"a5135b47-d2a7-45d5-ab3b-cb4404d1c00b\") " pod="openshift-multus/multus-additional-cni-plugins-5ddp8"
Jul 29 03:55:36.840805 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.840458    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/11451627-c54f-414d-a818-3acd21c7c829-host-cni-netd\") pod \"ovnkube-node-pznfg\" (UID: \"11451627-c54f-414d-a818-3acd21c7c829\") " pod="openshift-ovn-kubernetes/ovnkube-node-pznfg"
Jul 29 03:55:36.841789 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.840464    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a5135b47-d2a7-45d5-ab3b-cb4404d1c00b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5ddp8\" (UID: \"a5135b47-d2a7-45d5-ab3b-cb4404d1c00b\") " pod="openshift-multus/multus-additional-cni-plugins-5ddp8"
Jul 29 03:55:36.841789 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.840429    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/11451627-c54f-414d-a818-3acd21c7c829-var-lib-openvswitch\") pod \"ovnkube-node-pznfg\" (UID: \"11451627-c54f-414d-a818-3acd21c7c829\") " pod="openshift-ovn-kubernetes/ovnkube-node-pznfg"
Jul 29 03:55:36.841789 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.840391    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/34dcf361-8ea0-4b35-a259-0d0ab4ba5052-etc-kubernetes\") pod \"tuned-8p4vn\" (UID: \"34dcf361-8ea0-4b35-a259-0d0ab4ba5052\") " pod="openshift-cluster-node-tuning-operator/tuned-8p4vn"
Jul 29 03:55:36.841789 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.840489    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/179b4cec-34c3-4de7-9ff6-c9faee097c87-multus-cni-dir\") pod \"multus-dc668\" (UID: \"179b4cec-34c3-4de7-9ff6-c9faee097c87\") " pod="openshift-multus/multus-dc668"
Jul 29 03:55:36.841789 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.840510    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/179b4cec-34c3-4de7-9ff6-c9faee097c87-cnibin\") pod \"multus-dc668\" (UID: \"179b4cec-34c3-4de7-9ff6-c9faee097c87\") " pod="openshift-multus/multus-dc668"
Jul 29 03:55:36.841789 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.840514    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a5135b47-d2a7-45d5-ab3b-cb4404d1c00b-os-release\") pod \"multus-additional-cni-plugins-5ddp8\" (UID: \"a5135b47-d2a7-45d5-ab3b-cb4404d1c00b\") " pod="openshift-multus/multus-additional-cni-plugins-5ddp8"
Jul 29 03:55:36.841789 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.840593    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/179b4cec-34c3-4de7-9ff6-c9faee097c87-cnibin\") pod \"multus-dc668\" (UID: \"179b4cec-34c3-4de7-9ff6-c9faee097c87\") " pod="openshift-multus/multus-dc668"
Jul 29 03:55:36.841789 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.840616    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/179b4cec-34c3-4de7-9ff6-c9faee097c87-multus-cni-dir\") pod \"multus-dc668\" (UID: \"179b4cec-34c3-4de7-9ff6-c9faee097c87\") " pod="openshift-multus/multus-dc668"
Jul 29 03:55:36.841789 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.840628    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/179b4cec-34c3-4de7-9ff6-c9faee097c87-cni-binary-copy\") pod \"multus-dc668\" (UID: \"179b4cec-34c3-4de7-9ff6-c9faee097c87\") " pod="openshift-multus/multus-dc668"
Jul 29 03:55:36.841789 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.840643    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/11451627-c54f-414d-a818-3acd21c7c829-ovnkube-script-lib\") pod \"ovnkube-node-pznfg\" (UID: \"11451627-c54f-414d-a818-3acd21c7c829\") " pod="openshift-ovn-kubernetes/ovnkube-node-pznfg"
Jul 29 03:55:36.841789 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.840658    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64f62499-e7b7-4249-a1e4-636144783b55-metrics-certs\") pod \"network-metrics-daemon-5ddj8\" (UID: \"64f62499-e7b7-4249-a1e4-636144783b55\") " pod="openshift-multus/network-metrics-daemon-5ddj8"
Jul 29 03:55:36.841789 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.840683    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lj5r\" (UniqueName: \"kubernetes.io/projected/c00b9ff9-d986-4043-8403-c8d9a68596da-kube-api-access-4lj5r\") pod \"node-resolver-hc29h\" (UID: \"c00b9ff9-d986-4043-8403-c8d9a68596da\") " pod="openshift-dns/node-resolver-hc29h"
Jul 29 03:55:36.841789 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.840708    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcgsk\" (UniqueName: \"kubernetes.io/projected/a5135b47-d2a7-45d5-ab3b-cb4404d1c00b-kube-api-access-bcgsk\") pod \"multus-additional-cni-plugins-5ddp8\" (UID: \"a5135b47-d2a7-45d5-ab3b-cb4404d1c00b\") " pod="openshift-multus/multus-additional-cni-plugins-5ddp8"
Jul 29 03:55:36.841789 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.840732    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/11451627-c54f-414d-a818-3acd21c7c829-run-systemd\") pod \"ovnkube-node-pznfg\" (UID: \"11451627-c54f-414d-a818-3acd21c7c829\") " pod="openshift-ovn-kubernetes/ovnkube-node-pznfg"
Jul 29 03:55:36.841789 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.840755    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11451627-c54f-414d-a818-3acd21c7c829-host-run-ovn-kubernetes\") pod \"ovnkube-node-pznfg\" (UID: \"11451627-c54f-414d-a818-3acd21c7c829\") " pod="openshift-ovn-kubernetes/ovnkube-node-pznfg"
Jul 29 03:55:36.841789 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.840778    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/11451627-c54f-414d-a818-3acd21c7c829-ovnkube-config\") pod \"ovnkube-node-pznfg\" (UID: \"11451627-c54f-414d-a818-3acd21c7c829\") " pod="openshift-ovn-kubernetes/ovnkube-node-pznfg"
Jul 29 03:55:36.841789 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.840801    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/34dcf361-8ea0-4b35-a259-0d0ab4ba5052-etc-modprobe-d\") pod \"tuned-8p4vn\" (UID: \"34dcf361-8ea0-4b35-a259-0d0ab4ba5052\") " pod="openshift-cluster-node-tuning-operator/tuned-8p4vn"
Jul 29 03:55:36.842581 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.840842    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7b3ec617-26f1-4c28-aa08-2710238467ac-host\") pod \"node-ca-kkpf7\" (UID: \"7b3ec617-26f1-4c28-aa08-2710238467ac\") " pod="openshift-image-registry/node-ca-kkpf7"
Jul 29 03:55:36.842581 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.840866    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4e72f9a5-de78-4f59-b7b7-b6a1bd93c34c-registration-dir\") pod \"aws-ebs-csi-driver-node-8mn5m\" (UID: \"4e72f9a5-de78-4f59-b7b7-b6a1bd93c34c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8mn5m"
Jul 29 03:55:36.842581 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.840890    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/11451627-c54f-414d-a818-3acd21c7c829-run-ovn\") pod \"ovnkube-node-pznfg\" (UID: \"11451627-c54f-414d-a818-3acd21c7c829\") " pod="openshift-ovn-kubernetes/ovnkube-node-pznfg"
Jul 29 03:55:36.842581 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.840912    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82pqq\" (UniqueName: \"kubernetes.io/projected/11451627-c54f-414d-a818-3acd21c7c829-kube-api-access-82pqq\") pod \"ovnkube-node-pznfg\" (UID: \"11451627-c54f-414d-a818-3acd21c7c829\") " pod="openshift-ovn-kubernetes/ovnkube-node-pznfg"
Jul 29 03:55:36.842581 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.840939    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvmpk\" (UniqueName: \"kubernetes.io/projected/a0ccb95e-5e5c-4a10-9f35-6510b59e84aa-kube-api-access-hvmpk\") pod \"network-check-target-8222s\" (UID: \"a0ccb95e-5e5c-4a10-9f35-6510b59e84aa\") " pod="openshift-network-diagnostics/network-check-target-8222s"
Jul 29 03:55:36.842581 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.840963    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/179b4cec-34c3-4de7-9ff6-c9faee097c87-multus-socket-dir-parent\") pod \"multus-dc668\" (UID: \"179b4cec-34c3-4de7-9ff6-c9faee097c87\") " pod="openshift-multus/multus-dc668"
Jul 29 03:55:36.842581 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.840985    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/34dcf361-8ea0-4b35-a259-0d0ab4ba5052-etc-systemd\") pod \"tuned-8p4vn\" (UID: \"34dcf361-8ea0-4b35-a259-0d0ab4ba5052\") " pod="openshift-cluster-node-tuning-operator/tuned-8p4vn"
Jul 29 03:55:36.842581 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.841007    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/34dcf361-8ea0-4b35-a259-0d0ab4ba5052-etc-tuned\") pod \"tuned-8p4vn\" (UID: \"34dcf361-8ea0-4b35-a259-0d0ab4ba5052\") " pod="openshift-cluster-node-tuning-operator/tuned-8p4vn"
Jul 29 03:55:36.842581 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.841017    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a5135b47-d2a7-45d5-ab3b-cb4404d1c00b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5ddp8\" (UID: \"a5135b47-d2a7-45d5-ab3b-cb4404d1c00b\") " pod="openshift-multus/multus-additional-cni-plugins-5ddp8"
Jul 29 03:55:36.842581 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.841031    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d9e67808-ec21-42a2-bbaf-a28164326ccc-original-pull-secret\") pod \"global-pull-secret-syncer-vvzjp\" (UID: \"d9e67808-ec21-42a2-bbaf-a28164326ccc\") " pod="kube-system/global-pull-secret-syncer-vvzjp"
Jul 29 03:55:36.842581 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.841038    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d7d7563a-a7bd-477c-9f04-4a4ada546abd-iptables-alerter-script\") pod \"iptables-alerter-66gml\" (UID: \"d7d7563a-a7bd-477c-9f04-4a4ada546abd\") " pod="openshift-network-operator/iptables-alerter-66gml"
Jul 29 03:55:36.842581 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.841075    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a5135b47-d2a7-45d5-ab3b-cb4404d1c00b-cni-binary-copy\") pod \"multus-additional-cni-plugins-5ddp8\" (UID: \"a5135b47-d2a7-45d5-ab3b-cb4404d1c00b\") " pod="openshift-multus/multus-additional-cni-plugins-5ddp8"
Jul 29 03:55:36.842581 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.841103    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/179b4cec-34c3-4de7-9ff6-c9faee097c87-multus-daemon-config\") pod \"multus-dc668\" (UID: \"179b4cec-34c3-4de7-9ff6-c9faee097c87\") " pod="openshift-multus/multus-dc668"
Jul 29 03:55:36.842581 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.841125    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/179b4cec-34c3-4de7-9ff6-c9faee097c87-host-run-multus-certs\") pod \"multus-dc668\" (UID: \"179b4cec-34c3-4de7-9ff6-c9faee097c87\") " pod="openshift-multus/multus-dc668"
Jul 29 03:55:36.842581 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.841150    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/34dcf361-8ea0-4b35-a259-0d0ab4ba5052-etc-sysctl-conf\") pod \"tuned-8p4vn\" (UID: \"34dcf361-8ea0-4b35-a259-0d0ab4ba5052\") " pod="openshift-cluster-node-tuning-operator/tuned-8p4vn"
Jul 29 03:55:36.842581 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.841157    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7b3ec617-26f1-4c28-aa08-2710238467ac-host\") pod \"node-ca-kkpf7\" (UID: \"7b3ec617-26f1-4c28-aa08-2710238467ac\") " pod="openshift-image-registry/node-ca-kkpf7"
Jul 29 03:55:36.842581 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.841174    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/34dcf361-8ea0-4b35-a259-0d0ab4ba5052-run\") pod \"tuned-8p4vn\" (UID: \"34dcf361-8ea0-4b35-a259-0d0ab4ba5052\") " pod="openshift-cluster-node-tuning-operator/tuned-8p4vn"
Jul 29 03:55:36.843380 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.841198    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/34dcf361-8ea0-4b35-a259-0d0ab4ba5052-sys\") pod \"tuned-8p4vn\" (UID: \"34dcf361-8ea0-4b35-a259-0d0ab4ba5052\") " pod="openshift-cluster-node-tuning-operator/tuned-8p4vn"
Jul 29 03:55:36.843380 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.841214    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/179b4cec-34c3-4de7-9ff6-c9faee097c87-cni-binary-copy\") pod \"multus-dc668\" (UID: \"179b4cec-34c3-4de7-9ff6-c9faee097c87\") " pod="openshift-multus/multus-dc668"
Jul 29 03:55:36.843380 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.841221    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/34dcf361-8ea0-4b35-a259-0d0ab4ba5052-var-lib-kubelet\") pod \"tuned-8p4vn\" (UID: \"34dcf361-8ea0-4b35-a259-0d0ab4ba5052\") " pod="openshift-cluster-node-tuning-operator/tuned-8p4vn"
Jul 29 03:55:36.843380 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.841228    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/34dcf361-8ea0-4b35-a259-0d0ab4ba5052-etc-systemd\") pod \"tuned-8p4vn\" (UID: \"34dcf361-8ea0-4b35-a259-0d0ab4ba5052\") " pod="openshift-cluster-node-tuning-operator/tuned-8p4vn"
Jul 29 03:55:36.843380 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.841228    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/179b4cec-34c3-4de7-9ff6-c9faee097c87-multus-socket-dir-parent\") pod \"multus-dc668\" (UID: \"179b4cec-34c3-4de7-9ff6-c9faee097c87\") " pod="openshift-multus/multus-dc668"
Jul 29 03:55:36.843380 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:36.841126    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 03:55:36.843380 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:36.841128    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 03:55:36.843380 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.841246    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/11451627-c54f-414d-a818-3acd21c7c829-host-slash\") pod \"ovnkube-node-pznfg\" (UID: \"11451627-c54f-414d-a818-3acd21c7c829\") " pod="openshift-ovn-kubernetes/ovnkube-node-pznfg"
Jul 29 03:55:36.843380 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.841176    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4e72f9a5-de78-4f59-b7b7-b6a1bd93c34c-registration-dir\") pod \"aws-ebs-csi-driver-node-8mn5m\" (UID: \"4e72f9a5-de78-4f59-b7b7-b6a1bd93c34c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8mn5m"
Jul 29 03:55:36.843380 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.841355    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/34dcf361-8ea0-4b35-a259-0d0ab4ba5052-etc-modprobe-d\") pod \"tuned-8p4vn\" (UID: \"34dcf361-8ea0-4b35-a259-0d0ab4ba5052\") " pod="openshift-cluster-node-tuning-operator/tuned-8p4vn"
Jul 29 03:55:36.843380 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:36.841436    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9e67808-ec21-42a2-bbaf-a28164326ccc-original-pull-secret podName:d9e67808-ec21-42a2-bbaf-a28164326ccc nodeName:}" failed. No retries permitted until 2026-07-29 03:55:37.341415719 +0000 UTC m=+2.071546789 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d9e67808-ec21-42a2-bbaf-a28164326ccc-original-pull-secret") pod "global-pull-secret-syncer-vvzjp" (UID: "d9e67808-ec21-42a2-bbaf-a28164326ccc") : object "kube-system"/"original-pull-secret" not registered
Jul 29 03:55:36.843380 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.841549    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/34dcf361-8ea0-4b35-a259-0d0ab4ba5052-var-lib-kubelet\") pod \"tuned-8p4vn\" (UID: \"34dcf361-8ea0-4b35-a259-0d0ab4ba5052\") " pod="openshift-cluster-node-tuning-operator/tuned-8p4vn"
Jul 29 03:55:36.843380 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:36.841575    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64f62499-e7b7-4249-a1e4-636144783b55-metrics-certs podName:64f62499-e7b7-4249-a1e4-636144783b55 nodeName:}" failed. No retries permitted until 2026-07-29 03:55:37.341562139 +0000 UTC m=+2.071693187 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/64f62499-e7b7-4249-a1e4-636144783b55-metrics-certs") pod "network-metrics-daemon-5ddj8" (UID: "64f62499-e7b7-4249-a1e4-636144783b55") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 03:55:36.843380 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.841593    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11451627-c54f-414d-a818-3acd21c7c829-host-run-ovn-kubernetes\") pod \"ovnkube-node-pznfg\" (UID: \"11451627-c54f-414d-a818-3acd21c7c829\") " pod="openshift-ovn-kubernetes/ovnkube-node-pznfg"
Jul 29 03:55:36.843380 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.841607    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/11451627-c54f-414d-a818-3acd21c7c829-run-systemd\") pod \"ovnkube-node-pznfg\" (UID: \"11451627-c54f-414d-a818-3acd21c7c829\") " pod="openshift-ovn-kubernetes/ovnkube-node-pznfg"
Jul 29 03:55:36.843380 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.841638    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/11451627-c54f-414d-a818-3acd21c7c829-host-slash\") pod \"ovnkube-node-pznfg\" (UID: \"11451627-c54f-414d-a818-3acd21c7c829\") " pod="openshift-ovn-kubernetes/ovnkube-node-pznfg"
Jul 29 03:55:36.843380 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.841686    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/11451627-c54f-414d-a818-3acd21c7c829-run-ovn\") pod \"ovnkube-node-pznfg\" (UID: \"11451627-c54f-414d-a818-3acd21c7c829\") " pod="openshift-ovn-kubernetes/ovnkube-node-pznfg"
Jul 29 03:55:36.844164 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.841715    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/11451627-c54f-414d-a818-3acd21c7c829-ovnkube-config\") pod \"ovnkube-node-pznfg\" (UID: \"11451627-c54f-414d-a818-3acd21c7c829\") " pod="openshift-ovn-kubernetes/ovnkube-node-pznfg"
Jul 29 03:55:36.844164 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.841723    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/34dcf361-8ea0-4b35-a259-0d0ab4ba5052-run\") pod \"tuned-8p4vn\" (UID: \"34dcf361-8ea0-4b35-a259-0d0ab4ba5052\") " pod="openshift-cluster-node-tuning-operator/tuned-8p4vn"
Jul 29 03:55:36.844164 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.841738    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/179b4cec-34c3-4de7-9ff6-c9faee097c87-multus-daemon-config\") pod \"multus-dc668\" (UID: \"179b4cec-34c3-4de7-9ff6-c9faee097c87\") " pod="openshift-multus/multus-dc668"
Jul 29 03:55:36.844164 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.841771    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7b3ec617-26f1-4c28-aa08-2710238467ac-serviceca\") pod \"node-ca-kkpf7\" (UID: \"7b3ec617-26f1-4c28-aa08-2710238467ac\") " pod="openshift-image-registry/node-ca-kkpf7"
Jul 29 03:55:36.844164 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.841798    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4e72f9a5-de78-4f59-b7b7-b6a1bd93c34c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8mn5m\" (UID: \"4e72f9a5-de78-4f59-b7b7-b6a1bd93c34c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8mn5m"
Jul 29 03:55:36.844164 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.841843    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4e72f9a5-de78-4f59-b7b7-b6a1bd93c34c-sys-fs\") pod \"aws-ebs-csi-driver-node-8mn5m\" (UID: \"4e72f9a5-de78-4f59-b7b7-b6a1bd93c34c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8mn5m"
Jul 29 03:55:36.844164 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.841867    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d9e67808-ec21-42a2-bbaf-a28164326ccc-dbus\") pod \"global-pull-secret-syncer-vvzjp\" (UID: \"d9e67808-ec21-42a2-bbaf-a28164326ccc\") " pod="kube-system/global-pull-secret-syncer-vvzjp"
Jul 29 03:55:36.844164 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.841867    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/34dcf361-8ea0-4b35-a259-0d0ab4ba5052-etc-sysctl-conf\") pod \"tuned-8p4vn\" (UID: \"34dcf361-8ea0-4b35-a259-0d0ab4ba5052\") " pod="openshift-cluster-node-tuning-operator/tuned-8p4vn"
Jul 29 03:55:36.844164 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.841892    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d7d7563a-a7bd-477c-9f04-4a4ada546abd-host-slash\") pod \"iptables-alerter-66gml\" (UID: \"d7d7563a-a7bd-477c-9f04-4a4ada546abd\") " pod="openshift-network-operator/iptables-alerter-66gml"
Jul 29 03:55:36.844164 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.841916    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7252c41b-1992-4e77-b76d-8f036a55e064-konnectivity-ca\") pod \"konnectivity-agent-k5hmf\" (UID: \"7252c41b-1992-4e77-b76d-8f036a55e064\") " pod="kube-system/konnectivity-agent-k5hmf"
Jul 29 03:55:36.844164 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.841931    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4e72f9a5-de78-4f59-b7b7-b6a1bd93c34c-sys-fs\") pod \"aws-ebs-csi-driver-node-8mn5m\" (UID: \"4e72f9a5-de78-4f59-b7b7-b6a1bd93c34c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8mn5m"
Jul 29 03:55:36.844164 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.841941    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/179b4cec-34c3-4de7-9ff6-c9faee097c87-system-cni-dir\") pod \"multus-dc668\" (UID: \"179b4cec-34c3-4de7-9ff6-c9faee097c87\") " pod="openshift-multus/multus-dc668"
Jul 29 03:55:36.844164 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.841973    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmclr\" (UniqueName: \"kubernetes.io/projected/7b3ec617-26f1-4c28-aa08-2710238467ac-kube-api-access-nmclr\") pod \"node-ca-kkpf7\" (UID: \"7b3ec617-26f1-4c28-aa08-2710238467ac\") " pod="openshift-image-registry/node-ca-kkpf7"
Jul 29 03:55:36.844164 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.841979    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4e72f9a5-de78-4f59-b7b7-b6a1bd93c34c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8mn5m\" (UID: \"4e72f9a5-de78-4f59-b7b7-b6a1bd93c34c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8mn5m"
Jul 29 03:55:36.844164 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.841995    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/179b4cec-34c3-4de7-9ff6-c9faee097c87-host-var-lib-kubelet\") pod \"multus-dc668\" (UID: \"179b4cec-34c3-4de7-9ff6-c9faee097c87\") " pod="openshift-multus/multus-dc668"
Jul 29 03:55:36.844164 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.842024    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/11451627-c54f-414d-a818-3acd21c7c829-host-run-netns\") pod \"ovnkube-node-pznfg\" (UID: \"11451627-c54f-414d-a818-3acd21c7c829\") " pod="openshift-ovn-kubernetes/ovnkube-node-pznfg"
Jul 29 03:55:36.844164 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.842052    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/11451627-c54f-414d-a818-3acd21c7c829-run-openvswitch\") pod \"ovnkube-node-pznfg\" (UID: \"11451627-c54f-414d-a818-3acd21c7c829\") " pod="openshift-ovn-kubernetes/ovnkube-node-pznfg"
Jul 29 03:55:36.844680 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.842065    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a5135b47-d2a7-45d5-ab3b-cb4404d1c00b-cni-binary-copy\") pod \"multus-additional-cni-plugins-5ddp8\" (UID: \"a5135b47-d2a7-45d5-ab3b-cb4404d1c00b\") " pod="openshift-multus/multus-additional-cni-plugins-5ddp8"
Jul 29 03:55:36.844680 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.842074    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/179b4cec-34c3-4de7-9ff6-c9faee097c87-hostroot\") pod \"multus-dc668\" (UID: \"179b4cec-34c3-4de7-9ff6-c9faee097c87\") " pod="openshift-multus/multus-dc668"
Jul 29 03:55:36.844680 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.842107    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt4rs\" (UniqueName: \"kubernetes.io/projected/64f62499-e7b7-4249-a1e4-636144783b55-kube-api-access-wt4rs\") pod \"network-metrics-daemon-5ddj8\" (UID: \"64f62499-e7b7-4249-a1e4-636144783b55\") " pod="openshift-multus/network-metrics-daemon-5ddj8"
Jul 29 03:55:36.844680 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.842132    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/34dcf361-8ea0-4b35-a259-0d0ab4ba5052-host\") pod \"tuned-8p4vn\" (UID: \"34dcf361-8ea0-4b35-a259-0d0ab4ba5052\") " pod="openshift-cluster-node-tuning-operator/tuned-8p4vn"
Jul 29 03:55:36.844680 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.842135    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/179b4cec-34c3-4de7-9ff6-c9faee097c87-system-cni-dir\") pod \"multus-dc668\" (UID: \"179b4cec-34c3-4de7-9ff6-c9faee097c87\") " pod="openshift-multus/multus-dc668"
Jul 29 03:55:36.844680 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.842155    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/11451627-c54f-414d-a818-3acd21c7c829-host-kubelet\") pod \"ovnkube-node-pznfg\" (UID: \"11451627-c54f-414d-a818-3acd21c7c829\") " pod="openshift-ovn-kubernetes/ovnkube-node-pznfg"
Jul 29 03:55:36.844680 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.842179    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d9e67808-ec21-42a2-bbaf-a28164326ccc-kubelet-config\") pod \"global-pull-secret-syncer-vvzjp\" (UID: \"d9e67808-ec21-42a2-bbaf-a28164326ccc\") " pod="kube-system/global-pull-secret-syncer-vvzjp"
Jul 29 03:55:36.844680 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.842203    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfgm9\" (UniqueName: \"kubernetes.io/projected/d7d7563a-a7bd-477c-9f04-4a4ada546abd-kube-api-access-hfgm9\") pod \"iptables-alerter-66gml\" (UID: \"d7d7563a-a7bd-477c-9f04-4a4ada546abd\") " pod="openshift-network-operator/iptables-alerter-66gml"
Jul 29 03:55:36.844680 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.842229    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a5135b47-d2a7-45d5-ab3b-cb4404d1c00b-system-cni-dir\") pod \"multus-additional-cni-plugins-5ddp8\" (UID: \"a5135b47-d2a7-45d5-ab3b-cb4404d1c00b\") " pod="openshift-multus/multus-additional-cni-plugins-5ddp8"
Jul 29 03:55:36.844680 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.842254    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4e72f9a5-de78-4f59-b7b7-b6a1bd93c34c-socket-dir\") pod \"aws-ebs-csi-driver-node-8mn5m\" (UID: \"4e72f9a5-de78-4f59-b7b7-b6a1bd93c34c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8mn5m"
Jul 29 03:55:36.844680 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.842255    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7b3ec617-26f1-4c28-aa08-2710238467ac-serviceca\") pod \"node-ca-kkpf7\" (UID: \"7b3ec617-26f1-4c28-aa08-2710238467ac\") " pod="openshift-image-registry/node-ca-kkpf7"
Jul 29 03:55:36.844680 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.842276    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/11451627-c54f-414d-a818-3acd21c7c829-host-cni-bin\") pod \"ovnkube-node-pznfg\" (UID: \"11451627-c54f-414d-a818-3acd21c7c829\") " pod="openshift-ovn-kubernetes/ovnkube-node-pznfg"
Jul 29 03:55:36.844680 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.842327    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/11451627-c54f-414d-a818-3acd21c7c829-host-run-netns\") pod \"ovnkube-node-pznfg\" (UID: \"11451627-c54f-414d-a818-3acd21c7c829\") " pod="openshift-ovn-kubernetes/ovnkube-node-pznfg"
Jul 29 03:55:36.844680 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.842365    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/179b4cec-34c3-4de7-9ff6-c9faee097c87-hostroot\") pod \"multus-dc668\" (UID: \"179b4cec-34c3-4de7-9ff6-c9faee097c87\") " pod="openshift-multus/multus-dc668"
Jul 29 03:55:36.844680 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.842375    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/34dcf361-8ea0-4b35-a259-0d0ab4ba5052-sys\") pod \"tuned-8p4vn\" (UID: \"34dcf361-8ea0-4b35-a259-0d0ab4ba5052\") " pod="openshift-cluster-node-tuning-operator/tuned-8p4vn"
Jul 29 03:55:36.844680 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.842418    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/34dcf361-8ea0-4b35-a259-0d0ab4ba5052-host\") pod \"tuned-8p4vn\" (UID: \"34dcf361-8ea0-4b35-a259-0d0ab4ba5052\") " pod="openshift-cluster-node-tuning-operator/tuned-8p4vn"
Jul 29 03:55:36.844680 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.842456    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/11451627-c54f-414d-a818-3acd21c7c829-host-kubelet\") pod \"ovnkube-node-pznfg\" (UID: \"11451627-c54f-414d-a818-3acd21c7c829\") " pod="openshift-ovn-kubernetes/ovnkube-node-pznfg"
Jul 29 03:55:36.844680 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.842457    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7252c41b-1992-4e77-b76d-8f036a55e064-konnectivity-ca\") pod \"konnectivity-agent-k5hmf\" (UID: \"7252c41b-1992-4e77-b76d-8f036a55e064\") " pod="kube-system/konnectivity-agent-k5hmf"
Jul 29 03:55:36.845281 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.842507    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/179b4cec-34c3-4de7-9ff6-c9faee097c87-host-var-lib-kubelet\") pod \"multus-dc668\" (UID: \"179b4cec-34c3-4de7-9ff6-c9faee097c87\") " pod="openshift-multus/multus-dc668"
Jul 29 03:55:36.845281 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.842302    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/34dcf361-8ea0-4b35-a259-0d0ab4ba5052-lib-modules\") pod \"tuned-8p4vn\" (UID: \"34dcf361-8ea0-4b35-a259-0d0ab4ba5052\") " pod="openshift-cluster-node-tuning-operator/tuned-8p4vn"
Jul 29 03:55:36.845281 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.842520    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d9e67808-ec21-42a2-bbaf-a28164326ccc-dbus\") pod \"global-pull-secret-syncer-vvzjp\" (UID: \"d9e67808-ec21-42a2-bbaf-a28164326ccc\") " pod="kube-system/global-pull-secret-syncer-vvzjp"
Jul 29 03:55:36.845281 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.842523    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d9e67808-ec21-42a2-bbaf-a28164326ccc-kubelet-config\") pod \"global-pull-secret-syncer-vvzjp\" (UID: \"d9e67808-ec21-42a2-bbaf-a28164326ccc\") " pod="kube-system/global-pull-secret-syncer-vvzjp"
Jul 29 03:55:36.845281 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.842544    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/34dcf361-8ea0-4b35-a259-0d0ab4ba5052-tmp\") pod \"tuned-8p4vn\" (UID: \"34dcf361-8ea0-4b35-a259-0d0ab4ba5052\") " pod="openshift-cluster-node-tuning-operator/tuned-8p4vn"
Jul 29 03:55:36.845281 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.842586    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d7d7563a-a7bd-477c-9f04-4a4ada546abd-host-slash\") pod \"iptables-alerter-66gml\" (UID: \"d7d7563a-a7bd-477c-9f04-4a4ada546abd\") " pod="openshift-network-operator/iptables-alerter-66gml"
Jul 29 03:55:36.845281 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.842617    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9nwl\" (UniqueName: \"kubernetes.io/projected/34dcf361-8ea0-4b35-a259-0d0ab4ba5052-kube-api-access-b9nwl\") pod \"tuned-8p4vn\" (UID: \"34dcf361-8ea0-4b35-a259-0d0ab4ba5052\") " pod="openshift-cluster-node-tuning-operator/tuned-8p4vn"
Jul 29 03:55:36.845281 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.842618    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/11451627-c54f-414d-a818-3acd21c7c829-run-openvswitch\") pod \"ovnkube-node-pznfg\" (UID: \"11451627-c54f-414d-a818-3acd21c7c829\") " pod="openshift-ovn-kubernetes/ovnkube-node-pznfg"
Jul 29 03:55:36.845281 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.842644    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/34dcf361-8ea0-4b35-a259-0d0ab4ba5052-etc-sysctl-d\") pod \"tuned-8p4vn\" (UID: \"34dcf361-8ea0-4b35-a259-0d0ab4ba5052\") " pod="openshift-cluster-node-tuning-operator/tuned-8p4vn"
Jul 29 03:55:36.845281 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.842668    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a5135b47-d2a7-45d5-ab3b-cb4404d1c00b-cnibin\") pod \"multus-additional-cni-plugins-5ddp8\" (UID: \"a5135b47-d2a7-45d5-ab3b-cb4404d1c00b\") " pod="openshift-multus/multus-additional-cni-plugins-5ddp8"
Jul 29 03:55:36.845281 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.842714    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/11451627-c54f-414d-a818-3acd21c7c829-host-cni-bin\") pod \"ovnkube-node-pznfg\" (UID: \"11451627-c54f-414d-a818-3acd21c7c829\") " pod="openshift-ovn-kubernetes/ovnkube-node-pznfg"
Jul 29 03:55:36.845281 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.842715    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/179b4cec-34c3-4de7-9ff6-c9faee097c87-os-release\") pod \"multus-dc668\" (UID: \"179b4cec-34c3-4de7-9ff6-c9faee097c87\") " pod="openshift-multus/multus-dc668"
Jul 29 03:55:36.845281 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.842735    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4e72f9a5-de78-4f59-b7b7-b6a1bd93c34c-socket-dir\") pod \"aws-ebs-csi-driver-node-8mn5m\" (UID: \"4e72f9a5-de78-4f59-b7b7-b6a1bd93c34c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8mn5m"
Jul 29 03:55:36.845281 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.842748    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/179b4cec-34c3-4de7-9ff6-c9faee097c87-host-run-k8s-cni-cncf-io\") pod \"multus-dc668\" (UID: \"179b4cec-34c3-4de7-9ff6-c9faee097c87\") " pod="openshift-multus/multus-dc668"
Jul 29 03:55:36.845281 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.842768    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/179b4cec-34c3-4de7-9ff6-c9faee097c87-os-release\") pod \"multus-dc668\" (UID: \"179b4cec-34c3-4de7-9ff6-c9faee097c87\") " pod="openshift-multus/multus-dc668"
Jul 29 03:55:36.845281 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.842773    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/179b4cec-34c3-4de7-9ff6-c9faee097c87-host-run-netns\") pod \"multus-dc668\" (UID: \"179b4cec-34c3-4de7-9ff6-c9faee097c87\") " pod="openshift-multus/multus-dc668"
Jul 29 03:55:36.845281 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.842784    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a5135b47-d2a7-45d5-ab3b-cb4404d1c00b-system-cni-dir\") pod \"multus-additional-cni-plugins-5ddp8\" (UID: \"a5135b47-d2a7-45d5-ab3b-cb4404d1c00b\") " pod="openshift-multus/multus-additional-cni-plugins-5ddp8"
Jul 29 03:55:36.845281 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.842800    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/179b4cec-34c3-4de7-9ff6-c9faee097c87-multus-conf-dir\") pod \"multus-dc668\" (UID: \"179b4cec-34c3-4de7-9ff6-c9faee097c87\") " pod="openshift-multus/multus-dc668"
Jul 29 03:55:36.845808 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.842807    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/179b4cec-34c3-4de7-9ff6-c9faee097c87-host-run-netns\") pod \"multus-dc668\" (UID: \"179b4cec-34c3-4de7-9ff6-c9faee097c87\") " pod="openshift-multus/multus-dc668"
Jul 29 03:55:36.845808 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.842841    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/11451627-c54f-414d-a818-3acd21c7c829-etc-openvswitch\") pod \"ovnkube-node-pznfg\" (UID: \"11451627-c54f-414d-a818-3acd21c7c829\") " pod="openshift-ovn-kubernetes/ovnkube-node-pznfg"
Jul 29 03:55:36.845808 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.842865    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/179b4cec-34c3-4de7-9ff6-c9faee097c87-host-run-k8s-cni-cncf-io\") pod \"multus-dc668\" (UID: \"179b4cec-34c3-4de7-9ff6-c9faee097c87\") " pod="openshift-multus/multus-dc668"
Jul 29 03:55:36.845808 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.842868    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a5135b47-d2a7-45d5-ab3b-cb4404d1c00b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5ddp8\" (UID: \"a5135b47-d2a7-45d5-ab3b-cb4404d1c00b\") " pod="openshift-multus/multus-additional-cni-plugins-5ddp8"
Jul 29 03:55:36.845808 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.842896    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77l56\" (UniqueName: \"kubernetes.io/projected/179b4cec-34c3-4de7-9ff6-c9faee097c87-kube-api-access-77l56\") pod \"multus-dc668\" (UID: \"179b4cec-34c3-4de7-9ff6-c9faee097c87\") " pod="openshift-multus/multus-dc668"
Jul 29 03:55:36.845808 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.842902    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/179b4cec-34c3-4de7-9ff6-c9faee097c87-host-run-multus-certs\") pod \"multus-dc668\" (UID: \"179b4cec-34c3-4de7-9ff6-c9faee097c87\") " pod="openshift-multus/multus-dc668"
Jul 29 03:55:36.845808 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.842920    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/11451627-c54f-414d-a818-3acd21c7c829-node-log\") pod \"ovnkube-node-pznfg\" (UID: \"11451627-c54f-414d-a818-3acd21c7c829\") " pod="openshift-ovn-kubernetes/ovnkube-node-pznfg"
Jul 29 03:55:36.845808 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.842943    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/11451627-c54f-414d-a818-3acd21c7c829-env-overrides\") pod \"ovnkube-node-pznfg\" (UID: \"11451627-c54f-414d-a818-3acd21c7c829\") " pod="openshift-ovn-kubernetes/ovnkube-node-pznfg"
Jul 29 03:55:36.845808 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.842934    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/34dcf361-8ea0-4b35-a259-0d0ab4ba5052-lib-modules\") pod \"tuned-8p4vn\" (UID: \"34dcf361-8ea0-4b35-a259-0d0ab4ba5052\") " pod="openshift-cluster-node-tuning-operator/tuned-8p4vn"
Jul 29 03:55:36.845808 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.843010    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/179b4cec-34c3-4de7-9ff6-c9faee097c87-multus-conf-dir\") pod \"multus-dc668\" (UID: \"179b4cec-34c3-4de7-9ff6-c9faee097c87\") " pod="openshift-multus/multus-dc668"
Jul 29 03:55:36.845808 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.843047    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/34dcf361-8ea0-4b35-a259-0d0ab4ba5052-etc-sysctl-d\") pod \"tuned-8p4vn\" (UID: \"34dcf361-8ea0-4b35-a259-0d0ab4ba5052\") " pod="openshift-cluster-node-tuning-operator/tuned-8p4vn"
Jul 29 03:55:36.845808 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.843067    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/11451627-c54f-414d-a818-3acd21c7c829-etc-openvswitch\") pod \"ovnkube-node-pznfg\" (UID: \"11451627-c54f-414d-a818-3acd21c7c829\") " pod="openshift-ovn-kubernetes/ovnkube-node-pznfg"
Jul 29 03:55:36.845808 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.843109    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a5135b47-d2a7-45d5-ab3b-cb4404d1c00b-cnibin\") pod \"multus-additional-cni-plugins-5ddp8\" (UID: \"a5135b47-d2a7-45d5-ab3b-cb4404d1c00b\") " pod="openshift-multus/multus-additional-cni-plugins-5ddp8"
Jul 29 03:55:36.845808 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.843363    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/11451627-c54f-414d-a818-3acd21c7c829-node-log\") pod \"ovnkube-node-pznfg\" (UID: \"11451627-c54f-414d-a818-3acd21c7c829\") " pod="openshift-ovn-kubernetes/ovnkube-node-pznfg"
Jul 29 03:55:36.845808 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.843443    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/11451627-c54f-414d-a818-3acd21c7c829-env-overrides\") pod \"ovnkube-node-pznfg\" (UID: \"11451627-c54f-414d-a818-3acd21c7c829\") " pod="openshift-ovn-kubernetes/ovnkube-node-pznfg"
Jul 29 03:55:36.845808 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.843509    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a5135b47-d2a7-45d5-ab3b-cb4404d1c00b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5ddp8\" (UID: \"a5135b47-d2a7-45d5-ab3b-cb4404d1c00b\") " pod="openshift-multus/multus-additional-cni-plugins-5ddp8"
Jul 29 03:55:36.845808 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.843800    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/11451627-c54f-414d-a818-3acd21c7c829-ovn-node-metrics-cert\") pod \"ovnkube-node-pznfg\" (UID: \"11451627-c54f-414d-a818-3acd21c7c829\") " pod="openshift-ovn-kubernetes/ovnkube-node-pznfg"
Jul 29 03:55:36.845808 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.844127    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/34dcf361-8ea0-4b35-a259-0d0ab4ba5052-etc-tuned\") pod \"tuned-8p4vn\" (UID: \"34dcf361-8ea0-4b35-a259-0d0ab4ba5052\") " pod="openshift-cluster-node-tuning-operator/tuned-8p4vn"
Jul 29 03:55:36.846506 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.844185    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7252c41b-1992-4e77-b76d-8f036a55e064-agent-certs\") pod \"konnectivity-agent-k5hmf\" (UID: \"7252c41b-1992-4e77-b76d-8f036a55e064\") " pod="kube-system/konnectivity-agent-k5hmf"
Jul 29 03:55:36.846506 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.844682    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/34dcf361-8ea0-4b35-a259-0d0ab4ba5052-tmp\") pod \"tuned-8p4vn\" (UID: \"34dcf361-8ea0-4b35-a259-0d0ab4ba5052\") " pod="openshift-cluster-node-tuning-operator/tuned-8p4vn"
Jul 29 03:55:36.847846 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.847808    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbdtr\" (UniqueName: \"kubernetes.io/projected/4e72f9a5-de78-4f59-b7b7-b6a1bd93c34c-kube-api-access-xbdtr\") pod \"aws-ebs-csi-driver-node-8mn5m\" (UID: \"4e72f9a5-de78-4f59-b7b7-b6a1bd93c34c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8mn5m"
Jul 29 03:55:36.854042 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:36.854024    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 03:55:36.854119 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:36.854046    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 03:55:36.854119 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:36.854059    2570 projected.go:196] Error preparing data for projected volume kube-api-access-hvmpk for pod openshift-network-diagnostics/network-check-target-8222s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 03:55:36.854190 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:36.854122    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0ccb95e-5e5c-4a10-9f35-6510b59e84aa-kube-api-access-hvmpk podName:a0ccb95e-5e5c-4a10-9f35-6510b59e84aa nodeName:}" failed. No retries permitted until 2026-07-29 03:55:37.354099541 +0000 UTC m=+2.084230612 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-hvmpk" (UniqueName: "kubernetes.io/projected/a0ccb95e-5e5c-4a10-9f35-6510b59e84aa-kube-api-access-hvmpk") pod "network-check-target-8222s" (UID: "a0ccb95e-5e5c-4a10-9f35-6510b59e84aa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 03:55:36.855764 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.855740    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lj5r\" (UniqueName: \"kubernetes.io/projected/c00b9ff9-d986-4043-8403-c8d9a68596da-kube-api-access-4lj5r\") pod \"node-resolver-hc29h\" (UID: \"c00b9ff9-d986-4043-8403-c8d9a68596da\") " pod="openshift-dns/node-resolver-hc29h"
Jul 29 03:55:36.856717 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.856695    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcgsk\" (UniqueName: \"kubernetes.io/projected/a5135b47-d2a7-45d5-ab3b-cb4404d1c00b-kube-api-access-bcgsk\") pod \"multus-additional-cni-plugins-5ddp8\" (UID: \"a5135b47-d2a7-45d5-ab3b-cb4404d1c00b\") " pod="openshift-multus/multus-additional-cni-plugins-5ddp8"
Jul 29 03:55:36.857379 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.857355    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82pqq\" (UniqueName: \"kubernetes.io/projected/11451627-c54f-414d-a818-3acd21c7c829-kube-api-access-82pqq\") pod \"ovnkube-node-pznfg\" (UID: \"11451627-c54f-414d-a818-3acd21c7c829\") " pod="openshift-ovn-kubernetes/ovnkube-node-pznfg"
Jul 29 03:55:36.857601 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.857576    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77l56\" (UniqueName: \"kubernetes.io/projected/179b4cec-34c3-4de7-9ff6-c9faee097c87-kube-api-access-77l56\") pod \"multus-dc668\" (UID: \"179b4cec-34c3-4de7-9ff6-c9faee097c87\") " pod="openshift-multus/multus-dc668"
Jul 29 03:55:36.857772 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.857754    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfgm9\" (UniqueName: \"kubernetes.io/projected/d7d7563a-a7bd-477c-9f04-4a4ada546abd-kube-api-access-hfgm9\") pod \"iptables-alerter-66gml\" (UID: \"d7d7563a-a7bd-477c-9f04-4a4ada546abd\") " pod="openshift-network-operator/iptables-alerter-66gml"
Jul 29 03:55:36.858720 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.858693    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmclr\" (UniqueName: \"kubernetes.io/projected/7b3ec617-26f1-4c28-aa08-2710238467ac-kube-api-access-nmclr\") pod \"node-ca-kkpf7\" (UID: \"7b3ec617-26f1-4c28-aa08-2710238467ac\") " pod="openshift-image-registry/node-ca-kkpf7"
Jul 29 03:55:36.858720 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.858707    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt4rs\" (UniqueName: \"kubernetes.io/projected/64f62499-e7b7-4249-a1e4-636144783b55-kube-api-access-wt4rs\") pod \"network-metrics-daemon-5ddj8\" (UID: \"64f62499-e7b7-4249-a1e4-636144783b55\") " pod="openshift-multus/network-metrics-daemon-5ddj8"
Jul 29 03:55:36.858859 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.858847    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9nwl\" (UniqueName: \"kubernetes.io/projected/34dcf361-8ea0-4b35-a259-0d0ab4ba5052-kube-api-access-b9nwl\") pod \"tuned-8p4vn\" (UID: \"34dcf361-8ea0-4b35-a259-0d0ab4ba5052\") " pod="openshift-cluster-node-tuning-operator/tuned-8p4vn"
Jul 29 03:55:36.892108 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:36.892078    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fb5aae1a0930c70f1e97b9429e03a80.slice/crio-50064cdd71b56235ed820cbc2c755609a8441b851adb035ef6b8b714aff03bbd WatchSource:0}: Error finding container 50064cdd71b56235ed820cbc2c755609a8441b851adb035ef6b8b714aff03bbd: Status 404 returned error can't find the container with id 50064cdd71b56235ed820cbc2c755609a8441b851adb035ef6b8b714aff03bbd
Jul 29 03:55:36.892350 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:36.892329    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf49f38cf04909602874266f431d33228.slice/crio-1f3a020de3a5a01db6f6c0a227c6b44732d91cb3307904c2f2949c32d6610030 WatchSource:0}: Error finding container 1f3a020de3a5a01db6f6c0a227c6b44732d91cb3307904c2f2949c32d6610030: Status 404 returned error can't find the container with id 1f3a020de3a5a01db6f6c0a227c6b44732d91cb3307904c2f2949c32d6610030
Jul 29 03:55:36.897351 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.897338    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 03:55:36.977597 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.977544    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hc29h"
Jul 29 03:55:36.985170 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:36.985144    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc00b9ff9_d986_4043_8403_c8d9a68596da.slice/crio-45f4dcdd055d1237182581802caa7c45a444a12984219a72168415ebbf6d087c WatchSource:0}: Error finding container 45f4dcdd055d1237182581802caa7c45a444a12984219a72168415ebbf6d087c: Status 404 returned error can't find the container with id 45f4dcdd055d1237182581802caa7c45a444a12984219a72168415ebbf6d087c
Jul 29 03:55:36.999180 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:36.999161    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8p4vn"
Jul 29 03:55:37.004877 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:37.004863    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kkpf7"
Jul 29 03:55:37.005515 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:37.005491    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34dcf361_8ea0_4b35_a259_0d0ab4ba5052.slice/crio-270ead45af96755138abec1bdfdd3a9b693628bc5532e4bc3085a9353466b740 WatchSource:0}: Error finding container 270ead45af96755138abec1bdfdd3a9b693628bc5532e4bc3085a9353466b740: Status 404 returned error can't find the container with id 270ead45af96755138abec1bdfdd3a9b693628bc5532e4bc3085a9353466b740
Jul 29 03:55:37.012578 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:37.012559    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b3ec617_26f1_4c28_aa08_2710238467ac.slice/crio-ef7b4e8532eff06516755ec91a207533cbfdba345d603063c5ca5ceb52313ade WatchSource:0}: Error finding container ef7b4e8532eff06516755ec91a207533cbfdba345d603063c5ca5ceb52313ade: Status 404 returned error can't find the container with id ef7b4e8532eff06516755ec91a207533cbfdba345d603063c5ca5ceb52313ade
Jul 29 03:55:37.022436 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:37.022420    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pznfg"
Jul 29 03:55:37.028593 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:37.028573    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11451627_c54f_414d_a818_3acd21c7c829.slice/crio-fb69ce6ab95b3afd10ae456808a3cce38ba90018f92c139e15c267ea123516b5 WatchSource:0}: Error finding container fb69ce6ab95b3afd10ae456808a3cce38ba90018f92c139e15c267ea123516b5: Status 404 returned error can't find the container with id fb69ce6ab95b3afd10ae456808a3cce38ba90018f92c139e15c267ea123516b5
Jul 29 03:55:37.044330 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:37.044308    2570 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 03:55:37.047469 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:37.047454    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5ddp8"
Jul 29 03:55:37.052304 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:37.052289    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-66gml"
Jul 29 03:55:37.053700 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:37.053684    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5135b47_d2a7_45d5_ab3b_cb4404d1c00b.slice/crio-7e4a610ce007444f2586dde5949ee2f943e7b554eb4774f59d05fa0b4671e2d7 WatchSource:0}: Error finding container 7e4a610ce007444f2586dde5949ee2f943e7b554eb4774f59d05fa0b4671e2d7: Status 404 returned error can't find the container with id 7e4a610ce007444f2586dde5949ee2f943e7b554eb4774f59d05fa0b4671e2d7
Jul 29 03:55:37.058858 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:37.058839    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7d7563a_a7bd_477c_9f04_4a4ada546abd.slice/crio-03f80ea30754f6fb5adb4020bcff8a11bfcf122bb84663d2e520aeb6b5fb0c37 WatchSource:0}: Error finding container 03f80ea30754f6fb5adb4020bcff8a11bfcf122bb84663d2e520aeb6b5fb0c37: Status 404 returned error can't find the container with id 03f80ea30754f6fb5adb4020bcff8a11bfcf122bb84663d2e520aeb6b5fb0c37
Jul 29 03:55:37.081434 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:37.081416    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8mn5m"
Jul 29 03:55:37.088289 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:37.088271    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e72f9a5_de78_4f59_b7b7_b6a1bd93c34c.slice/crio-197c0c530ba85feed11e854489ed51039d2efa0a6954a32f3b1a89c28044876a WatchSource:0}: Error finding container 197c0c530ba85feed11e854489ed51039d2efa0a6954a32f3b1a89c28044876a: Status 404 returned error can't find the container with id 197c0c530ba85feed11e854489ed51039d2efa0a6954a32f3b1a89c28044876a
Jul 29 03:55:37.097087 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:37.097069    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-k5hmf"
Jul 29 03:55:37.102866 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:37.102848    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dc668"
Jul 29 03:55:37.103422 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:37.103404    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7252c41b_1992_4e77_b76d_8f036a55e064.slice/crio-0ac09351262c6a2d789c166225e5cc4203c06e30b9026b12cbc1fa21cccca59c WatchSource:0}: Error finding container 0ac09351262c6a2d789c166225e5cc4203c06e30b9026b12cbc1fa21cccca59c: Status 404 returned error can't find the container with id 0ac09351262c6a2d789c166225e5cc4203c06e30b9026b12cbc1fa21cccca59c
Jul 29 03:55:37.110336 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:55:37.110311    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod179b4cec_34c3_4de7_9ff6_c9faee097c87.slice/crio-52aa181d7519c621be6b7ef7d35f484038f90a84b4a6d7856267b7f7574f0252 WatchSource:0}: Error finding container 52aa181d7519c621be6b7ef7d35f484038f90a84b4a6d7856267b7f7574f0252: Status 404 returned error can't find the container with id 52aa181d7519c621be6b7ef7d35f484038f90a84b4a6d7856267b7f7574f0252
Jul 29 03:55:37.346189 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:37.346108    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64f62499-e7b7-4249-a1e4-636144783b55-metrics-certs\") pod \"network-metrics-daemon-5ddj8\" (UID: \"64f62499-e7b7-4249-a1e4-636144783b55\") " pod="openshift-multus/network-metrics-daemon-5ddj8"
Jul 29 03:55:37.346189 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:37.346164    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d9e67808-ec21-42a2-bbaf-a28164326ccc-original-pull-secret\") pod \"global-pull-secret-syncer-vvzjp\" (UID: \"d9e67808-ec21-42a2-bbaf-a28164326ccc\") " pod="kube-system/global-pull-secret-syncer-vvzjp"
Jul 29 03:55:37.346398 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:37.346296    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 03:55:37.346398 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:37.346352    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9e67808-ec21-42a2-bbaf-a28164326ccc-original-pull-secret podName:d9e67808-ec21-42a2-bbaf-a28164326ccc nodeName:}" failed. No retries permitted until 2026-07-29 03:55:38.346333151 +0000 UTC m=+3.076464213 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d9e67808-ec21-42a2-bbaf-a28164326ccc-original-pull-secret") pod "global-pull-secret-syncer-vvzjp" (UID: "d9e67808-ec21-42a2-bbaf-a28164326ccc") : object "kube-system"/"original-pull-secret" not registered
Jul 29 03:55:37.346748 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:37.346727    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 03:55:37.346907 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:37.346782    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64f62499-e7b7-4249-a1e4-636144783b55-metrics-certs podName:64f62499-e7b7-4249-a1e4-636144783b55 nodeName:}" failed. No retries permitted until 2026-07-29 03:55:38.346767028 +0000 UTC m=+3.076898080 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/64f62499-e7b7-4249-a1e4-636144783b55-metrics-certs") pod "network-metrics-daemon-5ddj8" (UID: "64f62499-e7b7-4249-a1e4-636144783b55") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 03:55:37.446704 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:37.446672    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvmpk\" (UniqueName: \"kubernetes.io/projected/a0ccb95e-5e5c-4a10-9f35-6510b59e84aa-kube-api-access-hvmpk\") pod \"network-check-target-8222s\" (UID: \"a0ccb95e-5e5c-4a10-9f35-6510b59e84aa\") " pod="openshift-network-diagnostics/network-check-target-8222s"
Jul 29 03:55:37.446910 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:37.446890    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 03:55:37.446973 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:37.446912    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 03:55:37.446973 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:37.446924    2570 projected.go:196] Error preparing data for projected volume kube-api-access-hvmpk for pod openshift-network-diagnostics/network-check-target-8222s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 03:55:37.447085 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:37.446977    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0ccb95e-5e5c-4a10-9f35-6510b59e84aa-kube-api-access-hvmpk podName:a0ccb95e-5e5c-4a10-9f35-6510b59e84aa nodeName:}" failed. No retries permitted until 2026-07-29 03:55:38.446959591 +0000 UTC m=+3.177090653 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-hvmpk" (UniqueName: "kubernetes.io/projected/a0ccb95e-5e5c-4a10-9f35-6510b59e84aa-kube-api-access-hvmpk") pod "network-check-target-8222s" (UID: "a0ccb95e-5e5c-4a10-9f35-6510b59e84aa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 03:55:37.684057 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:37.683963    2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 03:50:36 +0000 UTC" deadline="2028-04-17 09:18:31.297828038 +0000 UTC"
Jul 29 03:55:37.684057 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:37.684002    2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15077h22m53.613830505s"
Jul 29 03:55:37.793746 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:37.793552    2570 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 03:55:37.864440 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:37.864379    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pznfg" event={"ID":"11451627-c54f-414d-a818-3acd21c7c829","Type":"ContainerStarted","Data":"fb69ce6ab95b3afd10ae456808a3cce38ba90018f92c139e15c267ea123516b5"}
Jul 29 03:55:37.881068 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:37.881034    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kkpf7" event={"ID":"7b3ec617-26f1-4c28-aa08-2710238467ac","Type":"ContainerStarted","Data":"ef7b4e8532eff06516755ec91a207533cbfdba345d603063c5ca5ceb52313ade"}
Jul 29 03:55:37.899203 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:37.899174    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hc29h" event={"ID":"c00b9ff9-d986-4043-8403-c8d9a68596da","Type":"ContainerStarted","Data":"45f4dcdd055d1237182581802caa7c45a444a12984219a72168415ebbf6d087c"}
Jul 29 03:55:37.913142 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:37.913110    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-138.ec2.internal" event={"ID":"f49f38cf04909602874266f431d33228","Type":"ContainerStarted","Data":"1f3a020de3a5a01db6f6c0a227c6b44732d91cb3307904c2f2949c32d6610030"}
Jul 29 03:55:37.925313 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:37.925285    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-138.ec2.internal" event={"ID":"3fb5aae1a0930c70f1e97b9429e03a80","Type":"ContainerStarted","Data":"50064cdd71b56235ed820cbc2c755609a8441b851adb035ef6b8b714aff03bbd"}
Jul 29 03:55:37.949837 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:37.949696    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dc668" event={"ID":"179b4cec-34c3-4de7-9ff6-c9faee097c87","Type":"ContainerStarted","Data":"52aa181d7519c621be6b7ef7d35f484038f90a84b4a6d7856267b7f7574f0252"}
Jul 29 03:55:37.952641 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:37.952575    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-k5hmf" event={"ID":"7252c41b-1992-4e77-b76d-8f036a55e064","Type":"ContainerStarted","Data":"0ac09351262c6a2d789c166225e5cc4203c06e30b9026b12cbc1fa21cccca59c"}
Jul 29 03:55:37.986971 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:37.986938    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8mn5m" event={"ID":"4e72f9a5-de78-4f59-b7b7-b6a1bd93c34c","Type":"ContainerStarted","Data":"197c0c530ba85feed11e854489ed51039d2efa0a6954a32f3b1a89c28044876a"}
Jul 29 03:55:38.023850 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:38.023797    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-66gml" event={"ID":"d7d7563a-a7bd-477c-9f04-4a4ada546abd","Type":"ContainerStarted","Data":"03f80ea30754f6fb5adb4020bcff8a11bfcf122bb84663d2e520aeb6b5fb0c37"}
Jul 29 03:55:38.052773 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:38.052719    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5ddp8" event={"ID":"a5135b47-d2a7-45d5-ab3b-cb4404d1c00b","Type":"ContainerStarted","Data":"7e4a610ce007444f2586dde5949ee2f943e7b554eb4774f59d05fa0b4671e2d7"}
Jul 29 03:55:38.073248 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:38.073213    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8p4vn" event={"ID":"34dcf361-8ea0-4b35-a259-0d0ab4ba5052","Type":"ContainerStarted","Data":"270ead45af96755138abec1bdfdd3a9b693628bc5532e4bc3085a9353466b740"}
Jul 29 03:55:38.315945 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:38.315913    2570 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 03:55:38.356078 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:38.355967    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64f62499-e7b7-4249-a1e4-636144783b55-metrics-certs\") pod \"network-metrics-daemon-5ddj8\" (UID: \"64f62499-e7b7-4249-a1e4-636144783b55\") " pod="openshift-multus/network-metrics-daemon-5ddj8"
Jul 29 03:55:38.356078 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:38.356028    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d9e67808-ec21-42a2-bbaf-a28164326ccc-original-pull-secret\") pod \"global-pull-secret-syncer-vvzjp\" (UID: \"d9e67808-ec21-42a2-bbaf-a28164326ccc\") " pod="kube-system/global-pull-secret-syncer-vvzjp"
Jul 29 03:55:38.356303 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:38.356145    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 03:55:38.356303 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:38.356201    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9e67808-ec21-42a2-bbaf-a28164326ccc-original-pull-secret podName:d9e67808-ec21-42a2-bbaf-a28164326ccc nodeName:}" failed. No retries permitted until 2026-07-29 03:55:40.356184055 +0000 UTC m=+5.086315108 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d9e67808-ec21-42a2-bbaf-a28164326ccc-original-pull-secret") pod "global-pull-secret-syncer-vvzjp" (UID: "d9e67808-ec21-42a2-bbaf-a28164326ccc") : object "kube-system"/"original-pull-secret" not registered
Jul 29 03:55:38.356596 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:38.356577    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 03:55:38.356668 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:38.356630    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64f62499-e7b7-4249-a1e4-636144783b55-metrics-certs podName:64f62499-e7b7-4249-a1e4-636144783b55 nodeName:}" failed. No retries permitted until 2026-07-29 03:55:40.356614317 +0000 UTC m=+5.086745372 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/64f62499-e7b7-4249-a1e4-636144783b55-metrics-certs") pod "network-metrics-daemon-5ddj8" (UID: "64f62499-e7b7-4249-a1e4-636144783b55") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 03:55:38.456997 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:38.456966    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvmpk\" (UniqueName: \"kubernetes.io/projected/a0ccb95e-5e5c-4a10-9f35-6510b59e84aa-kube-api-access-hvmpk\") pod \"network-check-target-8222s\" (UID: \"a0ccb95e-5e5c-4a10-9f35-6510b59e84aa\") " pod="openshift-network-diagnostics/network-check-target-8222s"
Jul 29 03:55:38.457171 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:38.457148    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 03:55:38.457171 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:38.457171    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 03:55:38.457282 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:38.457184    2570 projected.go:196] Error preparing data for projected volume kube-api-access-hvmpk for pod openshift-network-diagnostics/network-check-target-8222s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 03:55:38.457282 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:38.457242    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0ccb95e-5e5c-4a10-9f35-6510b59e84aa-kube-api-access-hvmpk podName:a0ccb95e-5e5c-4a10-9f35-6510b59e84aa nodeName:}" failed. No retries permitted until 2026-07-29 03:55:40.457222499 +0000 UTC m=+5.187353548 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-hvmpk" (UniqueName: "kubernetes.io/projected/a0ccb95e-5e5c-4a10-9f35-6510b59e84aa-kube-api-access-hvmpk") pod "network-check-target-8222s" (UID: "a0ccb95e-5e5c-4a10-9f35-6510b59e84aa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 03:55:38.684604 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:38.684467    2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 03:50:36 +0000 UTC" deadline="2028-02-29 23:33:23.311456191 +0000 UTC"
Jul 29 03:55:38.684604 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:38.684524    2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13939h37m44.626937038s"
Jul 29 03:55:38.828975 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:38.828942    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ddj8"
Jul 29 03:55:38.829125 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:38.829090    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ddj8" podUID="64f62499-e7b7-4249-a1e4-636144783b55"
Jul 29 03:55:38.829509 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:38.829488    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vvzjp"
Jul 29 03:55:38.829613 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:38.829582    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vvzjp" podUID="d9e67808-ec21-42a2-bbaf-a28164326ccc"
Jul 29 03:55:38.829677 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:38.829639    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8222s"
Jul 29 03:55:38.829731 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:38.829706    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8222s" podUID="a0ccb95e-5e5c-4a10-9f35-6510b59e84aa"
Jul 29 03:55:39.075254 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:39.075180    2570 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 03:55:40.372547 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:40.372513    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64f62499-e7b7-4249-a1e4-636144783b55-metrics-certs\") pod \"network-metrics-daemon-5ddj8\" (UID: \"64f62499-e7b7-4249-a1e4-636144783b55\") " pod="openshift-multus/network-metrics-daemon-5ddj8"
Jul 29 03:55:40.372987 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:40.372579    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d9e67808-ec21-42a2-bbaf-a28164326ccc-original-pull-secret\") pod \"global-pull-secret-syncer-vvzjp\" (UID: \"d9e67808-ec21-42a2-bbaf-a28164326ccc\") " pod="kube-system/global-pull-secret-syncer-vvzjp"
Jul 29 03:55:40.372987 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:40.372727    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 03:55:40.372987 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:40.372784    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9e67808-ec21-42a2-bbaf-a28164326ccc-original-pull-secret podName:d9e67808-ec21-42a2-bbaf-a28164326ccc nodeName:}" failed. No retries permitted until 2026-07-29 03:55:44.37276581 +0000 UTC m=+9.102896877 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d9e67808-ec21-42a2-bbaf-a28164326ccc-original-pull-secret") pod "global-pull-secret-syncer-vvzjp" (UID: "d9e67808-ec21-42a2-bbaf-a28164326ccc") : object "kube-system"/"original-pull-secret" not registered
Jul 29 03:55:40.373209 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:40.373190    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 03:55:40.373277 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:40.373247    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64f62499-e7b7-4249-a1e4-636144783b55-metrics-certs podName:64f62499-e7b7-4249-a1e4-636144783b55 nodeName:}" failed. No retries permitted until 2026-07-29 03:55:44.373229957 +0000 UTC m=+9.103361007 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/64f62499-e7b7-4249-a1e4-636144783b55-metrics-certs") pod "network-metrics-daemon-5ddj8" (UID: "64f62499-e7b7-4249-a1e4-636144783b55") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 03:55:40.473655 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:40.473609    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvmpk\" (UniqueName: \"kubernetes.io/projected/a0ccb95e-5e5c-4a10-9f35-6510b59e84aa-kube-api-access-hvmpk\") pod \"network-check-target-8222s\" (UID: \"a0ccb95e-5e5c-4a10-9f35-6510b59e84aa\") " pod="openshift-network-diagnostics/network-check-target-8222s"
Jul 29 03:55:40.473839 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:40.473786    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 03:55:40.473839 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:40.473806    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 03:55:40.473839 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:40.473835    2570 projected.go:196] Error preparing data for projected volume kube-api-access-hvmpk for pod openshift-network-diagnostics/network-check-target-8222s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 03:55:40.474001 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:40.473890    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0ccb95e-5e5c-4a10-9f35-6510b59e84aa-kube-api-access-hvmpk podName:a0ccb95e-5e5c-4a10-9f35-6510b59e84aa nodeName:}" failed. No retries permitted until 2026-07-29 03:55:44.473872633 +0000 UTC m=+9.204003684 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-hvmpk" (UniqueName: "kubernetes.io/projected/a0ccb95e-5e5c-4a10-9f35-6510b59e84aa-kube-api-access-hvmpk") pod "network-check-target-8222s" (UID: "a0ccb95e-5e5c-4a10-9f35-6510b59e84aa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 03:55:40.828936 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:40.828813    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vvzjp"
Jul 29 03:55:40.829088 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:40.828975    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vvzjp" podUID="d9e67808-ec21-42a2-bbaf-a28164326ccc"
Jul 29 03:55:40.829088 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:40.829039    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8222s"
Jul 29 03:55:40.829195 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:40.829090    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8222s" podUID="a0ccb95e-5e5c-4a10-9f35-6510b59e84aa"
Jul 29 03:55:40.829195 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:40.829148    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ddj8"
Jul 29 03:55:40.829296 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:40.829241    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ddj8" podUID="64f62499-e7b7-4249-a1e4-636144783b55"
Jul 29 03:55:42.829663 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:42.828879    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8222s"
Jul 29 03:55:42.829663 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:42.829004    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8222s" podUID="a0ccb95e-5e5c-4a10-9f35-6510b59e84aa"
Jul 29 03:55:42.829663 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:42.829391    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ddj8"
Jul 29 03:55:42.829663 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:42.829481    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ddj8" podUID="64f62499-e7b7-4249-a1e4-636144783b55"
Jul 29 03:55:42.829663 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:42.829543    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vvzjp"
Jul 29 03:55:42.829663 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:42.829611    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vvzjp" podUID="d9e67808-ec21-42a2-bbaf-a28164326ccc"
Jul 29 03:55:44.403092 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:44.403053    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64f62499-e7b7-4249-a1e4-636144783b55-metrics-certs\") pod \"network-metrics-daemon-5ddj8\" (UID: \"64f62499-e7b7-4249-a1e4-636144783b55\") " pod="openshift-multus/network-metrics-daemon-5ddj8"
Jul 29 03:55:44.403533 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:44.403126    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d9e67808-ec21-42a2-bbaf-a28164326ccc-original-pull-secret\") pod \"global-pull-secret-syncer-vvzjp\" (UID: \"d9e67808-ec21-42a2-bbaf-a28164326ccc\") " pod="kube-system/global-pull-secret-syncer-vvzjp"
Jul 29 03:55:44.403533 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:44.403239    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 03:55:44.403533 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:44.403243    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 03:55:44.403533 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:44.403314    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64f62499-e7b7-4249-a1e4-636144783b55-metrics-certs podName:64f62499-e7b7-4249-a1e4-636144783b55 nodeName:}" failed. No retries permitted until 2026-07-29 03:55:52.403293792 +0000 UTC m=+17.133424853 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/64f62499-e7b7-4249-a1e4-636144783b55-metrics-certs") pod "network-metrics-daemon-5ddj8" (UID: "64f62499-e7b7-4249-a1e4-636144783b55") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 03:55:44.403533 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:44.403333    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9e67808-ec21-42a2-bbaf-a28164326ccc-original-pull-secret podName:d9e67808-ec21-42a2-bbaf-a28164326ccc nodeName:}" failed. No retries permitted until 2026-07-29 03:55:52.40332354 +0000 UTC m=+17.133454591 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d9e67808-ec21-42a2-bbaf-a28164326ccc-original-pull-secret") pod "global-pull-secret-syncer-vvzjp" (UID: "d9e67808-ec21-42a2-bbaf-a28164326ccc") : object "kube-system"/"original-pull-secret" not registered
Jul 29 03:55:44.503810 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:44.503772    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvmpk\" (UniqueName: \"kubernetes.io/projected/a0ccb95e-5e5c-4a10-9f35-6510b59e84aa-kube-api-access-hvmpk\") pod \"network-check-target-8222s\" (UID: \"a0ccb95e-5e5c-4a10-9f35-6510b59e84aa\") " pod="openshift-network-diagnostics/network-check-target-8222s"
Jul 29 03:55:44.503999 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:44.503939    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 03:55:44.503999 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:44.503965    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 03:55:44.503999 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:44.503978    2570 projected.go:196] Error preparing data for projected volume kube-api-access-hvmpk for pod openshift-network-diagnostics/network-check-target-8222s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 03:55:44.504133 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:44.504045    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0ccb95e-5e5c-4a10-9f35-6510b59e84aa-kube-api-access-hvmpk podName:a0ccb95e-5e5c-4a10-9f35-6510b59e84aa nodeName:}" failed. No retries permitted until 2026-07-29 03:55:52.504026445 +0000 UTC m=+17.234157499 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-hvmpk" (UniqueName: "kubernetes.io/projected/a0ccb95e-5e5c-4a10-9f35-6510b59e84aa-kube-api-access-hvmpk") pod "network-check-target-8222s" (UID: "a0ccb95e-5e5c-4a10-9f35-6510b59e84aa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 03:55:44.830168 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:44.829321    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8222s"
Jul 29 03:55:44.830168 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:44.829446    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8222s" podUID="a0ccb95e-5e5c-4a10-9f35-6510b59e84aa"
Jul 29 03:55:44.830168 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:44.829840    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ddj8"
Jul 29 03:55:44.830168 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:44.829952    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ddj8" podUID="64f62499-e7b7-4249-a1e4-636144783b55"
Jul 29 03:55:44.830168 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:44.830006    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vvzjp"
Jul 29 03:55:44.830168 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:44.830079    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vvzjp" podUID="d9e67808-ec21-42a2-bbaf-a28164326ccc"
Jul 29 03:55:46.829215 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:46.829099    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vvzjp"
Jul 29 03:55:46.829215 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:46.829160    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8222s"
Jul 29 03:55:46.829215 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:46.829160    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ddj8"
Jul 29 03:55:46.829739 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:46.829253    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vvzjp" podUID="d9e67808-ec21-42a2-bbaf-a28164326ccc"
Jul 29 03:55:46.829739 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:46.829346    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8222s" podUID="a0ccb95e-5e5c-4a10-9f35-6510b59e84aa"
Jul 29 03:55:46.829739 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:46.829439    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ddj8" podUID="64f62499-e7b7-4249-a1e4-636144783b55"
Jul 29 03:55:48.829373 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:48.829351    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vvzjp"
Jul 29 03:55:48.829373 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:48.829363    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8222s"
Jul 29 03:55:48.829725 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:48.829383    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ddj8"
Jul 29 03:55:48.829725 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:48.829457    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vvzjp" podUID="d9e67808-ec21-42a2-bbaf-a28164326ccc"
Jul 29 03:55:48.829725 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:48.829563    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8222s" podUID="a0ccb95e-5e5c-4a10-9f35-6510b59e84aa"
Jul 29 03:55:48.829725 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:48.829635    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ddj8" podUID="64f62499-e7b7-4249-a1e4-636144783b55"
Jul 29 03:55:50.829487 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:50.829303    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ddj8"
Jul 29 03:55:50.829903 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:50.829335    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vvzjp"
Jul 29 03:55:50.829903 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:50.829580    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ddj8" podUID="64f62499-e7b7-4249-a1e4-636144783b55"
Jul 29 03:55:50.829903 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:50.829340    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8222s"
Jul 29 03:55:50.829903 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:50.829628    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vvzjp" podUID="d9e67808-ec21-42a2-bbaf-a28164326ccc"
Jul 29 03:55:50.829903 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:50.829711    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8222s" podUID="a0ccb95e-5e5c-4a10-9f35-6510b59e84aa"
Jul 29 03:55:52.463447 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:52.463411    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64f62499-e7b7-4249-a1e4-636144783b55-metrics-certs\") pod \"network-metrics-daemon-5ddj8\" (UID: \"64f62499-e7b7-4249-a1e4-636144783b55\") " pod="openshift-multus/network-metrics-daemon-5ddj8"
Jul 29 03:55:52.463864 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:52.463465    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d9e67808-ec21-42a2-bbaf-a28164326ccc-original-pull-secret\") pod \"global-pull-secret-syncer-vvzjp\" (UID: \"d9e67808-ec21-42a2-bbaf-a28164326ccc\") " pod="kube-system/global-pull-secret-syncer-vvzjp"
Jul 29 03:55:52.463864 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:52.463587    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 03:55:52.463864 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:52.463586    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 03:55:52.463864 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:52.463646    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9e67808-ec21-42a2-bbaf-a28164326ccc-original-pull-secret podName:d9e67808-ec21-42a2-bbaf-a28164326ccc nodeName:}" failed. No retries permitted until 2026-07-29 03:56:08.463631835 +0000 UTC m=+33.193762883 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d9e67808-ec21-42a2-bbaf-a28164326ccc-original-pull-secret") pod "global-pull-secret-syncer-vvzjp" (UID: "d9e67808-ec21-42a2-bbaf-a28164326ccc") : object "kube-system"/"original-pull-secret" not registered
Jul 29 03:55:52.463864 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:52.463660    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64f62499-e7b7-4249-a1e4-636144783b55-metrics-certs podName:64f62499-e7b7-4249-a1e4-636144783b55 nodeName:}" failed. No retries permitted until 2026-07-29 03:56:08.463653083 +0000 UTC m=+33.193784130 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/64f62499-e7b7-4249-a1e4-636144783b55-metrics-certs") pod "network-metrics-daemon-5ddj8" (UID: "64f62499-e7b7-4249-a1e4-636144783b55") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 03:55:52.564345 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:52.564314    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvmpk\" (UniqueName: \"kubernetes.io/projected/a0ccb95e-5e5c-4a10-9f35-6510b59e84aa-kube-api-access-hvmpk\") pod \"network-check-target-8222s\" (UID: \"a0ccb95e-5e5c-4a10-9f35-6510b59e84aa\") " pod="openshift-network-diagnostics/network-check-target-8222s"
Jul 29 03:55:52.564537 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:52.564481    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 03:55:52.564537 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:52.564505    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 03:55:52.564537 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:52.564515    2570 projected.go:196] Error preparing data for projected volume kube-api-access-hvmpk for pod openshift-network-diagnostics/network-check-target-8222s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 03:55:52.564666 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:52.564570    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0ccb95e-5e5c-4a10-9f35-6510b59e84aa-kube-api-access-hvmpk podName:a0ccb95e-5e5c-4a10-9f35-6510b59e84aa nodeName:}" failed. No retries permitted until 2026-07-29 03:56:08.564552413 +0000 UTC m=+33.294683480 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-hvmpk" (UniqueName: "kubernetes.io/projected/a0ccb95e-5e5c-4a10-9f35-6510b59e84aa-kube-api-access-hvmpk") pod "network-check-target-8222s" (UID: "a0ccb95e-5e5c-4a10-9f35-6510b59e84aa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 03:55:52.829529 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:52.829453    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vvzjp"
Jul 29 03:55:52.829529 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:52.829489    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ddj8"
Jul 29 03:55:52.829529 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:52.829498    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8222s"
Jul 29 03:55:52.829799 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:52.829575    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vvzjp" podUID="d9e67808-ec21-42a2-bbaf-a28164326ccc"
Jul 29 03:55:52.829799 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:52.829630    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ddj8" podUID="64f62499-e7b7-4249-a1e4-636144783b55"
Jul 29 03:55:52.829799 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:52.829731    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8222s" podUID="a0ccb95e-5e5c-4a10-9f35-6510b59e84aa"
Jul 29 03:55:54.828770 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:54.828740    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ddj8"
Jul 29 03:55:54.829231 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:54.828876    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vvzjp"
Jul 29 03:55:54.829231 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:54.828885    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ddj8" podUID="64f62499-e7b7-4249-a1e4-636144783b55"
Jul 29 03:55:54.829231 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:54.828904    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8222s"
Jul 29 03:55:54.829231 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:54.828978    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vvzjp" podUID="d9e67808-ec21-42a2-bbaf-a28164326ccc"
Jul 29 03:55:54.829231 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:54.829051    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8222s" podUID="a0ccb95e-5e5c-4a10-9f35-6510b59e84aa"
Jul 29 03:55:56.828738 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:56.828704    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8222s"
Jul 29 03:55:56.829052 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:56.828750    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ddj8"
Jul 29 03:55:56.829052 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:56.828803    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vvzjp"
Jul 29 03:55:56.829052 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:56.828831    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8222s" podUID="a0ccb95e-5e5c-4a10-9f35-6510b59e84aa"
Jul 29 03:55:56.829052 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:56.828889    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vvzjp" podUID="d9e67808-ec21-42a2-bbaf-a28164326ccc"
Jul 29 03:55:56.829052 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:56.828950    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ddj8" podUID="64f62499-e7b7-4249-a1e4-636144783b55"
Jul 29 03:55:58.119839 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:58.119606    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dc668" event={"ID":"179b4cec-34c3-4de7-9ff6-c9faee097c87","Type":"ContainerStarted","Data":"f03e2b04af5fc1176b87dd37c2a98f1ae0f06d0af336cdd350ebbe670b2579d6"}
Jul 29 03:55:58.121090 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:58.121067    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-k5hmf" event={"ID":"7252c41b-1992-4e77-b76d-8f036a55e064","Type":"ContainerStarted","Data":"3d9c53daa888bd13b986baa92dbea60d830bc8fd365cf10bea16a76627cf873f"}
Jul 29 03:55:58.122509 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:58.122479    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8mn5m" event={"ID":"4e72f9a5-de78-4f59-b7b7-b6a1bd93c34c","Type":"ContainerStarted","Data":"b61acda9fe719062505aaa99a586aeeca4557e0bd335c663b27c62d3fe60d5ce"}
Jul 29 03:55:58.123746 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:58.123727    2570 generic.go:356] "Generic (PLEG): container finished" podID="a5135b47-d2a7-45d5-ab3b-cb4404d1c00b" containerID="21506c452d0aff8286a808b02eb7947e8e4e1ed38a6e4be66d5ac6b4bbb110ac" exitCode=0
Jul 29 03:55:58.123848 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:58.123784    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5ddp8" event={"ID":"a5135b47-d2a7-45d5-ab3b-cb4404d1c00b","Type":"ContainerDied","Data":"21506c452d0aff8286a808b02eb7947e8e4e1ed38a6e4be66d5ac6b4bbb110ac"}
Jul 29 03:55:58.125303 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:58.125273    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8p4vn" event={"ID":"34dcf361-8ea0-4b35-a259-0d0ab4ba5052","Type":"ContainerStarted","Data":"72a0ac6f2a448aa0fae0cbefa0ffb26f2f344f0bf33c408429ef7e5f93d19f07"}
Jul 29 03:55:58.128165 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:58.128147    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pznfg" event={"ID":"11451627-c54f-414d-a818-3acd21c7c829","Type":"ContainerStarted","Data":"b95e72a5ccc72d9b71b8736a972f533b2eed5b3723bf3ac364151edb3de4c2d0"}
Jul 29 03:55:58.128244 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:58.128206    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pznfg" event={"ID":"11451627-c54f-414d-a818-3acd21c7c829","Type":"ContainerStarted","Data":"02edd1fcf1aaab5a3b1ed7488c19e0c77a2c668527965a9ca04250799f710faf"}
Jul 29 03:55:58.128303 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:58.128243    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pznfg" event={"ID":"11451627-c54f-414d-a818-3acd21c7c829","Type":"ContainerStarted","Data":"45d093c949cec298270768e104b20a389adfe65edde189bb9f57806445278103"}
Jul 29 03:55:58.128352 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:58.128312    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pznfg" event={"ID":"11451627-c54f-414d-a818-3acd21c7c829","Type":"ContainerStarted","Data":"670bacc13389b296b31e22acfdc2ba122c4b99d811ee1b4baf1618aaf19ad55c"}
Jul 29 03:55:58.128387 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:58.128362    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pznfg" event={"ID":"11451627-c54f-414d-a818-3acd21c7c829","Type":"ContainerStarted","Data":"27e994f56d63f1a6b5cd71e5c0fb6c25cd7e8e68557eccabfad7307cab9a60d5"}
Jul 29 03:55:58.128427 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:58.128410    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pznfg" event={"ID":"11451627-c54f-414d-a818-3acd21c7c829","Type":"ContainerStarted","Data":"a48105e7cdc872a88291abd64a10d53ab3928a4a6162b9e4e28fb25f89d6beeb"}
Jul 29 03:55:58.129558 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:58.129535    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kkpf7" event={"ID":"7b3ec617-26f1-4c28-aa08-2710238467ac","Type":"ContainerStarted","Data":"4e93b8c755be8e733a21503f8565adc82e450e4982f99203ae5dab4a974e2ba1"}
Jul 29 03:55:58.130846 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:58.130808    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hc29h" event={"ID":"c00b9ff9-d986-4043-8403-c8d9a68596da","Type":"ContainerStarted","Data":"b38b17635652d86d38b7cdc76f9dc413c496e6e4f444670c09c04b06d7df2aae"}
Jul 29 03:55:58.132093 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:58.132074    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-138.ec2.internal" event={"ID":"f49f38cf04909602874266f431d33228","Type":"ContainerStarted","Data":"9999327b301cdfa432f24bbf66c47a4081be7923adfa705a719b57f8b058fc8e"}
Jul 29 03:55:58.133316 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:58.133294    2570 generic.go:356] "Generic (PLEG): container finished" podID="3fb5aae1a0930c70f1e97b9429e03a80" containerID="bad0777b04caaaa9e4d3bb3c350d13b06db84823f4ce2405195b9efadcf6bb35" exitCode=0
Jul 29 03:55:58.133381 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:58.133325    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-138.ec2.internal" event={"ID":"3fb5aae1a0930c70f1e97b9429e03a80","Type":"ContainerDied","Data":"bad0777b04caaaa9e4d3bb3c350d13b06db84823f4ce2405195b9efadcf6bb35"}
Jul 29 03:55:58.137141 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:58.137105    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-dc668" podStartSLOduration=3.184290524 podStartE2EDuration="23.137094073s" totalImagesPullingTime="19.952803549s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 03:55:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:55:58.136621145 +0000 UTC m=+22.866752229" watchObservedRunningTime="2026-07-29 03:55:58.137094073 +0000 UTC m=+22.867225142"
Jul 29 03:55:58.165912 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:58.165862    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-138.ec2.internal" podStartSLOduration=22.165847872 podStartE2EDuration="22.165847872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 03:55:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:55:58.165654952 +0000 UTC m=+22.895786022" watchObservedRunningTime="2026-07-29 03:55:58.165847872 +0000 UTC m=+22.895978941"
Jul 29 03:55:58.180061 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:58.179983    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-hc29h" podStartSLOduration=3.11622043 podStartE2EDuration="23.17996881s" totalImagesPullingTime="20.06374838s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 03:55:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:55:58.179911766 +0000 UTC m=+22.910042846" watchObservedRunningTime="2026-07-29 03:55:58.17996881 +0000 UTC m=+22.910099879"
Jul 29 03:55:58.198257 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:58.198209    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-8p4vn" podStartSLOduration=3.151643057 podStartE2EDuration="23.198186859s" totalImagesPullingTime="20.046543802s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 03:55:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:55:58.197845567 +0000 UTC m=+22.927976657" watchObservedRunningTime="2026-07-29 03:55:58.198186859 +0000 UTC m=+22.928317930"
Jul 29 03:55:58.217875 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:58.217809    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-kkpf7" podStartSLOduration=3.182564809 podStartE2EDuration="23.217788984s" totalImagesPullingTime="20.035224175s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 03:55:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:55:58.21735691 +0000 UTC m=+22.947487981" watchObservedRunningTime="2026-07-29 03:55:58.217788984 +0000 UTC m=+22.947920099"
Jul 29 03:55:58.260308 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:58.260028    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-k5hmf" podStartSLOduration=3.314124616 podStartE2EDuration="23.260008812s" totalImagesPullingTime="19.945884196s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 03:55:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:55:58.259164553 +0000 UTC m=+22.989295624" watchObservedRunningTime="2026-07-29 03:55:58.260008812 +0000 UTC m=+22.990139883"
Jul 29 03:55:58.698961 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:58.698941    2570 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 29 03:55:58.708642 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:58.708571    2570 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-29T03:55:58.698957664Z","UUID":"636b6c6e-a479-49c5-afb4-94785ff8e44d","Handler":null,"Name":"","Endpoint":""}
Jul 29 03:55:58.709889 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:58.709870    2570 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 29 03:55:58.709976 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:58.709905    2570 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 29 03:55:58.828870 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:58.828840    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ddj8"
Jul 29 03:55:58.828870 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:58.828871    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8222s"
Jul 29 03:55:58.829076 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:58.828840    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vvzjp"
Jul 29 03:55:58.829076 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:58.828956    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ddj8" podUID="64f62499-e7b7-4249-a1e4-636144783b55"
Jul 29 03:55:58.829076 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:58.829007    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vvzjp" podUID="d9e67808-ec21-42a2-bbaf-a28164326ccc"
Jul 29 03:55:58.829184 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:55:58.829075    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8222s" podUID="a0ccb95e-5e5c-4a10-9f35-6510b59e84aa"
Jul 29 03:55:59.137516 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:59.137468    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8mn5m" event={"ID":"4e72f9a5-de78-4f59-b7b7-b6a1bd93c34c","Type":"ContainerStarted","Data":"ab62df7abb7045a32628bc224159aefe4216bafc138fe5a032b284c7d84fe1de"}
Jul 29 03:55:59.139196 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:59.139165    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-66gml" event={"ID":"d7d7563a-a7bd-477c-9f04-4a4ada546abd","Type":"ContainerStarted","Data":"1336ff3c78db3a855132c747932897535326de4cb426167f57274e937db0917c"}
Jul 29 03:55:59.141305 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:59.141278    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-138.ec2.internal" event={"ID":"3fb5aae1a0930c70f1e97b9429e03a80","Type":"ContainerStarted","Data":"0dc55512400c4108c1ab90c308319dd3bcc683affb03e4039e50109d74214d71"}
Jul 29 03:55:59.154023 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:59.153978    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-66gml" podStartSLOduration=4.186915742 podStartE2EDuration="24.153964196s" totalImagesPullingTime="19.967048454s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 03:55:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:55:59.153437351 +0000 UTC m=+23.883568439" watchObservedRunningTime="2026-07-29 03:55:59.153964196 +0000 UTC m=+23.884095268"
Jul 29 03:55:59.168209 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:55:59.168169    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-138.ec2.internal" podStartSLOduration=23.168155812 podStartE2EDuration="23.168155812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 03:55:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:55:59.167792647 +0000 UTC m=+23.897923716" watchObservedRunningTime="2026-07-29 03:55:59.168155812 +0000 UTC m=+23.898286882"
Jul 29 03:56:00.148118 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:00.148085    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8mn5m" event={"ID":"4e72f9a5-de78-4f59-b7b7-b6a1bd93c34c","Type":"ContainerStarted","Data":"c99e3ef8e3b0ee96ec7a727e3a6c97d31acca62692f7f708f15277ac21266f3f"}
Jul 29 03:56:00.153975 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:00.153932    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pznfg" event={"ID":"11451627-c54f-414d-a818-3acd21c7c829","Type":"ContainerStarted","Data":"fd8f38afba62a12b775c1e29a5c5a7955cd185960ae5513a4553eea38ca54e42"}
Jul 29 03:56:00.165718 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:00.165660    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8mn5m" podStartSLOduration=3.005366511 podStartE2EDuration="25.165646038s" totalImagesPullingTime="22.160279527s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 03:55:35 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:56:00.164345405 +0000 UTC m=+24.894476476" watchObservedRunningTime="2026-07-29 03:56:00.165646038 +0000 UTC m=+24.895777138"
Jul 29 03:56:00.249135 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:00.249106    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-k5hmf"
Jul 29 03:56:00.249607 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:00.249584    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-k5hmf"
Jul 29 03:56:00.829133 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:00.829100    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vvzjp"
Jul 29 03:56:00.829290 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:00.829202    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8222s"
Jul 29 03:56:00.829290 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:56:00.829212    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vvzjp" podUID="d9e67808-ec21-42a2-bbaf-a28164326ccc"
Jul 29 03:56:00.829290 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:00.829228    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ddj8"
Jul 29 03:56:00.829395 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:56:00.829292    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8222s" podUID="a0ccb95e-5e5c-4a10-9f35-6510b59e84aa"
Jul 29 03:56:00.829395 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:56:00.829352    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ddj8" podUID="64f62499-e7b7-4249-a1e4-636144783b55"
Jul 29 03:56:01.156129 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:01.156099    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-k5hmf"
Jul 29 03:56:01.156631 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:01.156599    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-k5hmf"
Jul 29 03:56:02.829465 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:02.829290    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vvzjp"
Jul 29 03:56:02.829895 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:02.829317    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8222s"
Jul 29 03:56:02.829895 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:56:02.829598    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vvzjp" podUID="d9e67808-ec21-42a2-bbaf-a28164326ccc"
Jul 29 03:56:02.829895 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:02.829323    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ddj8"
Jul 29 03:56:02.829895 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:56:02.829622    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8222s" podUID="a0ccb95e-5e5c-4a10-9f35-6510b59e84aa"
Jul 29 03:56:02.829895 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:56:02.829727    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ddj8" podUID="64f62499-e7b7-4249-a1e4-636144783b55"
Jul 29 03:56:03.161875 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:03.161843    2570 generic.go:356] "Generic (PLEG): container finished" podID="a5135b47-d2a7-45d5-ab3b-cb4404d1c00b" containerID="bf0dc8e8e871d6b145c62a525b563ea54e45dc1b088b718da4e645843a70522d" exitCode=0
Jul 29 03:56:03.162027 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:03.161933    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5ddp8" event={"ID":"a5135b47-d2a7-45d5-ab3b-cb4404d1c00b","Type":"ContainerDied","Data":"bf0dc8e8e871d6b145c62a525b563ea54e45dc1b088b718da4e645843a70522d"}
Jul 29 03:56:03.165654 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:03.165621    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pznfg" event={"ID":"11451627-c54f-414d-a818-3acd21c7c829","Type":"ContainerStarted","Data":"2334df9488f1e26cd9a794bf64f032ac50b1e1a62e7d4b900fd3da8a8e5294a5"}
Jul 29 03:56:03.165798 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:03.165763    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pznfg"
Jul 29 03:56:03.208152 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:03.208114    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-pznfg" podStartSLOduration=8.100675189 podStartE2EDuration="28.208100354s" totalImagesPullingTime="20.107425165s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 03:55:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:56:03.206275285 +0000 UTC m=+27.936406367" watchObservedRunningTime="2026-07-29 03:56:03.208100354 +0000 UTC m=+27.938231424"
Jul 29 03:56:04.168428 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:04.168393    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pznfg"
Jul 29 03:56:04.169075 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:04.168471    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pznfg"
Jul 29 03:56:04.185630 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:04.185605    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pznfg"
Jul 29 03:56:04.185750 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:04.185704    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pznfg"
Jul 29 03:56:04.644296 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:04.644106    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8222s"]
Jul 29 03:56:04.644504 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:04.644407    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8222s"
Jul 29 03:56:04.644582 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:56:04.644518    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8222s" podUID="a0ccb95e-5e5c-4a10-9f35-6510b59e84aa"
Jul 29 03:56:04.647834 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:04.647779    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5ddj8"]
Jul 29 03:56:04.647948 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:04.647925    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ddj8"
Jul 29 03:56:04.648045 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:56:04.648024    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ddj8" podUID="64f62499-e7b7-4249-a1e4-636144783b55"
Jul 29 03:56:04.648457 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:04.648424    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vvzjp"]
Jul 29 03:56:04.648533 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:04.648521    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vvzjp"
Jul 29 03:56:04.648632 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:56:04.648612    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vvzjp" podUID="d9e67808-ec21-42a2-bbaf-a28164326ccc"
Jul 29 03:56:05.172048 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:05.172019    2570 generic.go:356] "Generic (PLEG): container finished" podID="a5135b47-d2a7-45d5-ab3b-cb4404d1c00b" containerID="8fb0db55993b59748e060905674fc99e20b839a196d31024f3baa08666c37be0" exitCode=0
Jul 29 03:56:05.172599 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:05.172105    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5ddp8" event={"ID":"a5135b47-d2a7-45d5-ab3b-cb4404d1c00b","Type":"ContainerDied","Data":"8fb0db55993b59748e060905674fc99e20b839a196d31024f3baa08666c37be0"}
Jul 29 03:56:05.829976 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:05.829840    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8222s"
Jul 29 03:56:05.829976 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:56:05.829938    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8222s" podUID="a0ccb95e-5e5c-4a10-9f35-6510b59e84aa"
Jul 29 03:56:05.830197 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:05.829917    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ddj8"
Jul 29 03:56:05.830197 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:56:05.830110    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ddj8" podUID="64f62499-e7b7-4249-a1e4-636144783b55"
Jul 29 03:56:06.808813 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:06.808789    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hc29h_c00b9ff9-d986-4043-8403-c8d9a68596da/dns-node-resolver/0.log"
Jul 29 03:56:06.829186 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:06.829163    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vvzjp"
Jul 29 03:56:06.829330 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:56:06.829266    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vvzjp" podUID="d9e67808-ec21-42a2-bbaf-a28164326ccc"
Jul 29 03:56:07.178386 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:07.178297    2570 generic.go:356] "Generic (PLEG): container finished" podID="a5135b47-d2a7-45d5-ab3b-cb4404d1c00b" containerID="0d3352ae1aaf8652e3f0563133e9317891895eab47dc94833b2a19c688521b7e" exitCode=0
Jul 29 03:56:07.178386 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:07.178344    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5ddp8" event={"ID":"a5135b47-d2a7-45d5-ab3b-cb4404d1c00b","Type":"ContainerDied","Data":"0d3352ae1aaf8652e3f0563133e9317891895eab47dc94833b2a19c688521b7e"}
Jul 29 03:56:07.787632 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:07.787596    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kkpf7_7b3ec617-26f1-4c28-aa08-2710238467ac/node-ca/0.log"
Jul 29 03:56:07.828937 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:07.828904    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ddj8"
Jul 29 03:56:07.829351 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:56:07.829031    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ddj8" podUID="64f62499-e7b7-4249-a1e4-636144783b55"
Jul 29 03:56:07.829351 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:07.829087    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8222s"
Jul 29 03:56:07.829351 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:56:07.829202    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8222s" podUID="a0ccb95e-5e5c-4a10-9f35-6510b59e84aa"
Jul 29 03:56:08.483404 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:08.483319    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64f62499-e7b7-4249-a1e4-636144783b55-metrics-certs\") pod \"network-metrics-daemon-5ddj8\" (UID: \"64f62499-e7b7-4249-a1e4-636144783b55\") " pod="openshift-multus/network-metrics-daemon-5ddj8"
Jul 29 03:56:08.483404 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:08.483382    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d9e67808-ec21-42a2-bbaf-a28164326ccc-original-pull-secret\") pod \"global-pull-secret-syncer-vvzjp\" (UID: \"d9e67808-ec21-42a2-bbaf-a28164326ccc\") " pod="kube-system/global-pull-secret-syncer-vvzjp"
Jul 29 03:56:08.483618 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:56:08.483493    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 03:56:08.483618 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:56:08.483558    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64f62499-e7b7-4249-a1e4-636144783b55-metrics-certs podName:64f62499-e7b7-4249-a1e4-636144783b55 nodeName:}" failed. No retries permitted until 2026-07-29 03:56:40.48354421 +0000 UTC m=+65.213675258 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/64f62499-e7b7-4249-a1e4-636144783b55-metrics-certs") pod "network-metrics-daemon-5ddj8" (UID: "64f62499-e7b7-4249-a1e4-636144783b55") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 03:56:08.483618 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:56:08.483493    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 03:56:08.483618 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:56:08.483614    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9e67808-ec21-42a2-bbaf-a28164326ccc-original-pull-secret podName:d9e67808-ec21-42a2-bbaf-a28164326ccc nodeName:}" failed. No retries permitted until 2026-07-29 03:56:40.483603635 +0000 UTC m=+65.213734689 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d9e67808-ec21-42a2-bbaf-a28164326ccc-original-pull-secret") pod "global-pull-secret-syncer-vvzjp" (UID: "d9e67808-ec21-42a2-bbaf-a28164326ccc") : object "kube-system"/"original-pull-secret" not registered
Jul 29 03:56:08.584128 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:08.584089    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvmpk\" (UniqueName: \"kubernetes.io/projected/a0ccb95e-5e5c-4a10-9f35-6510b59e84aa-kube-api-access-hvmpk\") pod \"network-check-target-8222s\" (UID: \"a0ccb95e-5e5c-4a10-9f35-6510b59e84aa\") " pod="openshift-network-diagnostics/network-check-target-8222s"
Jul 29 03:56:08.584287 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:56:08.584267    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 03:56:08.584339 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:56:08.584297    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 03:56:08.584339 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:56:08.584310    2570 projected.go:196] Error preparing data for projected volume kube-api-access-hvmpk for pod openshift-network-diagnostics/network-check-target-8222s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 03:56:08.584416 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:56:08.584363    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0ccb95e-5e5c-4a10-9f35-6510b59e84aa-kube-api-access-hvmpk podName:a0ccb95e-5e5c-4a10-9f35-6510b59e84aa nodeName:}" failed. No retries permitted until 2026-07-29 03:56:40.584350179 +0000 UTC m=+65.314481227 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-hvmpk" (UniqueName: "kubernetes.io/projected/a0ccb95e-5e5c-4a10-9f35-6510b59e84aa-kube-api-access-hvmpk") pod "network-check-target-8222s" (UID: "a0ccb95e-5e5c-4a10-9f35-6510b59e84aa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 03:56:08.828626 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:08.828546    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vvzjp"
Jul 29 03:56:08.828789 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:56:08.828664    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vvzjp" podUID="d9e67808-ec21-42a2-bbaf-a28164326ccc"
Jul 29 03:56:09.828943 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:09.828904    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8222s"
Jul 29 03:56:09.829485 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:09.828914    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ddj8"
Jul 29 03:56:09.829485 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:56:09.829033    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8222s" podUID="a0ccb95e-5e5c-4a10-9f35-6510b59e84aa"
Jul 29 03:56:09.829485 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:56:09.829135    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ddj8" podUID="64f62499-e7b7-4249-a1e4-636144783b55"
Jul 29 03:56:10.828952 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:10.828914    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vvzjp"
Jul 29 03:56:10.829351 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:56:10.829053    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vvzjp" podUID="d9e67808-ec21-42a2-bbaf-a28164326ccc"
Jul 29 03:56:11.190030 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:11.189864    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5ddp8" event={"ID":"a5135b47-d2a7-45d5-ab3b-cb4404d1c00b","Type":"ContainerStarted","Data":"af19959a6f1213f8e8e612ac3b80c2ce6294cea46e6f3df9c4b84d856e9a2954"}
Jul 29 03:56:11.828528 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:11.828494    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ddj8"
Jul 29 03:56:11.828716 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:56:11.828602    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ddj8" podUID="64f62499-e7b7-4249-a1e4-636144783b55"
Jul 29 03:56:11.828716 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:11.828656    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8222s"
Jul 29 03:56:11.828860 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:56:11.828749    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8222s" podUID="a0ccb95e-5e5c-4a10-9f35-6510b59e84aa"
Jul 29 03:56:12.194019 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:12.193989    2570 generic.go:356] "Generic (PLEG): container finished" podID="a5135b47-d2a7-45d5-ab3b-cb4404d1c00b" containerID="af19959a6f1213f8e8e612ac3b80c2ce6294cea46e6f3df9c4b84d856e9a2954" exitCode=0
Jul 29 03:56:12.194420 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:12.194055    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5ddp8" event={"ID":"a5135b47-d2a7-45d5-ab3b-cb4404d1c00b","Type":"ContainerDied","Data":"af19959a6f1213f8e8e612ac3b80c2ce6294cea46e6f3df9c4b84d856e9a2954"}
Jul 29 03:56:12.828675 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:12.828648    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vvzjp"
Jul 29 03:56:12.828873 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:56:12.828766    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vvzjp" podUID="d9e67808-ec21-42a2-bbaf-a28164326ccc"
Jul 29 03:56:13.198586 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:13.198547    2570 generic.go:356] "Generic (PLEG): container finished" podID="a5135b47-d2a7-45d5-ab3b-cb4404d1c00b" containerID="a167285f9de2ff4594c077c139837146fabfa984728b18e571cc606b79d942d3" exitCode=0
Jul 29 03:56:13.199060 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:13.198619    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5ddp8" event={"ID":"a5135b47-d2a7-45d5-ab3b-cb4404d1c00b","Type":"ContainerDied","Data":"a167285f9de2ff4594c077c139837146fabfa984728b18e571cc606b79d942d3"}
Jul 29 03:56:13.828912 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:13.828887    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8222s"
Jul 29 03:56:13.829092 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:56:13.828976    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8222s" podUID="a0ccb95e-5e5c-4a10-9f35-6510b59e84aa"
Jul 29 03:56:13.829092 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:13.829003    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ddj8"
Jul 29 03:56:13.829092 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:56:13.829057    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ddj8" podUID="64f62499-e7b7-4249-a1e4-636144783b55"
Jul 29 03:56:14.203311 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:14.203275    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5ddp8" event={"ID":"a5135b47-d2a7-45d5-ab3b-cb4404d1c00b","Type":"ContainerStarted","Data":"82d35dcf19e2aa9574b466732094f880d94e6eee60e99c9958145d1a6ec31598"}
Jul 29 03:56:14.226444 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:14.226398    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-5ddp8" podStartSLOduration=8.735855204 podStartE2EDuration="39.226383761s" totalImagesPullingTime="30.490528557s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 03:55:35 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:56:14.2244178 +0000 UTC m=+38.954548864" watchObservedRunningTime="2026-07-29 03:56:14.226383761 +0000 UTC m=+38.956514832"
Jul 29 03:56:14.829134 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:14.829096    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vvzjp"
Jul 29 03:56:14.829304 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:56:14.829209    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vvzjp" podUID="d9e67808-ec21-42a2-bbaf-a28164326ccc"
Jul 29 03:56:15.829330 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:15.829298    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8222s"
Jul 29 03:56:15.829704 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:56:15.829380    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8222s" podUID="a0ccb95e-5e5c-4a10-9f35-6510b59e84aa"
Jul 29 03:56:15.829704 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:15.829464    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ddj8"
Jul 29 03:56:15.829704 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:56:15.829566    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ddj8" podUID="64f62499-e7b7-4249-a1e4-636144783b55"
Jul 29 03:56:16.828782 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:16.828748    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vvzjp"
Jul 29 03:56:16.828956 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:56:16.828861    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vvzjp" podUID="d9e67808-ec21-42a2-bbaf-a28164326ccc"
Jul 29 03:56:17.828453 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:17.828419    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ddj8"
Jul 29 03:56:17.828809 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:56:17.828535    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ddj8" podUID="64f62499-e7b7-4249-a1e4-636144783b55"
Jul 29 03:56:17.828809 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:17.828586    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8222s"
Jul 29 03:56:17.828809 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:56:17.828670    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8222s" podUID="a0ccb95e-5e5c-4a10-9f35-6510b59e84aa"
Jul 29 03:56:18.829419 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:18.829392    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vvzjp"
Jul 29 03:56:18.829772 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:56:18.829488    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vvzjp" podUID="d9e67808-ec21-42a2-bbaf-a28164326ccc"
Jul 29 03:56:19.829129 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:19.829100    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ddj8"
Jul 29 03:56:19.829290 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:56:19.829209    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ddj8" podUID="64f62499-e7b7-4249-a1e4-636144783b55"
Jul 29 03:56:19.829334 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:19.829289    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8222s"
Jul 29 03:56:19.829418 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:56:19.829398    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8222s" podUID="a0ccb95e-5e5c-4a10-9f35-6510b59e84aa"
Jul 29 03:56:20.828859 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:20.828813    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vvzjp"
Jul 29 03:56:20.829217 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:56:20.828920    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vvzjp" podUID="d9e67808-ec21-42a2-bbaf-a28164326ccc"
Jul 29 03:56:21.829050 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:21.829018    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ddj8"
Jul 29 03:56:21.829469 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:21.829067    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8222s"
Jul 29 03:56:21.829469 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:56:21.829151    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ddj8" podUID="64f62499-e7b7-4249-a1e4-636144783b55"
Jul 29 03:56:21.829469 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:56:21.829253    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8222s" podUID="a0ccb95e-5e5c-4a10-9f35-6510b59e84aa"
Jul 29 03:56:22.828628 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:22.828597    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vvzjp"
Jul 29 03:56:22.828765 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:56:22.828697    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vvzjp" podUID="d9e67808-ec21-42a2-bbaf-a28164326ccc"
Jul 29 03:56:23.828745 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:23.828711    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8222s"
Jul 29 03:56:23.829199 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:23.828714    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ddj8"
Jul 29 03:56:23.829199 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:56:23.828804    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8222s" podUID="a0ccb95e-5e5c-4a10-9f35-6510b59e84aa"
Jul 29 03:56:23.829199 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:56:23.828940    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ddj8" podUID="64f62499-e7b7-4249-a1e4-636144783b55"
Jul 29 03:56:24.828811 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:24.828784    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vvzjp"
Jul 29 03:56:24.829187 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:56:24.828917    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vvzjp" podUID="d9e67808-ec21-42a2-bbaf-a28164326ccc"
Jul 29 03:56:25.829899 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:25.829865    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8222s"
Jul 29 03:56:25.830459 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:56:25.829954    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8222s" podUID="a0ccb95e-5e5c-4a10-9f35-6510b59e84aa"
Jul 29 03:56:25.830459 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:25.830036    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ddj8"
Jul 29 03:56:25.830459 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:56:25.830132    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ddj8" podUID="64f62499-e7b7-4249-a1e4-636144783b55"
Jul 29 03:56:26.828702 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:26.828668    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vvzjp"
Jul 29 03:56:26.828897 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:56:26.828784    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vvzjp" podUID="d9e67808-ec21-42a2-bbaf-a28164326ccc"
Jul 29 03:56:27.831434 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:27.831404    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ddj8"
Jul 29 03:56:27.831759 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:27.831445    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8222s"
Jul 29 03:56:27.831759 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:56:27.831521    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ddj8" podUID="64f62499-e7b7-4249-a1e4-636144783b55"
Jul 29 03:56:27.831759 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:56:27.831602    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8222s" podUID="a0ccb95e-5e5c-4a10-9f35-6510b59e84aa"
Jul 29 03:56:28.101636 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:28.101609    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-138.ec2.internal" event="NodeReady"
Jul 29 03:56:28.101806 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:28.101730    2570 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 29 03:56:28.150560 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:28.150529    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-nhdcg"]
Jul 29 03:56:28.170177 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:28.170156    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-mf7ck"]
Jul 29 03:56:28.170287 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:28.170253    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nhdcg"
Jul 29 03:56:28.172905 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:28.172881    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 29 03:56:28.173028 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:28.172952    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 29 03:56:28.173028 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:28.172960    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-6rjhj\""
Jul 29 03:56:28.173028 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:28.172977    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 29 03:56:28.189993 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:28.189974    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nhdcg"]
Jul 29 03:56:28.190091 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:28.190002    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mf7ck"]
Jul 29 03:56:28.190091 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:28.190077    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mf7ck"
Jul 29 03:56:28.193093 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:28.193067    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 29 03:56:28.193093 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:28.193081    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kz6jn\""
Jul 29 03:56:28.193256 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:28.193101    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 29 03:56:28.193256 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:28.193114    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 29 03:56:28.193256 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:28.193215    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 29 03:56:28.246413 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:28.246387    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-9rmhw"]
Jul 29 03:56:28.266064 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:28.266042    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9rmhw"]
Jul 29 03:56:28.266190 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:28.266130    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9rmhw"
Jul 29 03:56:28.268953 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:28.268932    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 29 03:56:28.268953 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:28.268945    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 29 03:56:28.269110 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:28.268950    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fhgfp\""
Jul 29 03:56:28.328010 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:28.327989    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/55fa3eb7-80e7-4b52-bdbe-766ae9a5b93f-data-volume\") pod \"insights-runtime-extractor-mf7ck\" (UID: \"55fa3eb7-80e7-4b52-bdbe-766ae9a5b93f\") " pod="openshift-insights/insights-runtime-extractor-mf7ck"
Jul 29 03:56:28.328127 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:28.328014    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/55fa3eb7-80e7-4b52-bdbe-766ae9a5b93f-crio-socket\") pod \"insights-runtime-extractor-mf7ck\" (UID: \"55fa3eb7-80e7-4b52-bdbe-766ae9a5b93f\") " pod="openshift-insights/insights-runtime-extractor-mf7ck"
Jul 29 03:56:28.328127 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:28.328036    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/55fa3eb7-80e7-4b52-bdbe-766ae9a5b93f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mf7ck\" (UID: \"55fa3eb7-80e7-4b52-bdbe-766ae9a5b93f\") " pod="openshift-insights/insights-runtime-extractor-mf7ck"
Jul 29 03:56:28.328127 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:28.328054    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nzbq\" (UniqueName: \"kubernetes.io/projected/00b8825f-8f3b-4da7-929c-df23b76f32d8-kube-api-access-6nzbq\") pod \"ingress-canary-nhdcg\" (UID: \"00b8825f-8f3b-4da7-929c-df23b76f32d8\") " pod="openshift-ingress-canary/ingress-canary-nhdcg"
Jul 29 03:56:28.328127 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:28.328080    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmls4\" (UniqueName: \"kubernetes.io/projected/55fa3eb7-80e7-4b52-bdbe-766ae9a5b93f-kube-api-access-mmls4\") pod \"insights-runtime-extractor-mf7ck\" (UID: \"55fa3eb7-80e7-4b52-bdbe-766ae9a5b93f\") " pod="openshift-insights/insights-runtime-extractor-mf7ck"
Jul 29 03:56:28.328127 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:28.328107    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/55fa3eb7-80e7-4b52-bdbe-766ae9a5b93f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mf7ck\" (UID: \"55fa3eb7-80e7-4b52-bdbe-766ae9a5b93f\") " pod="openshift-insights/insights-runtime-extractor-mf7ck"
Jul 29 03:56:28.328298 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:28.328134    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/00b8825f-8f3b-4da7-929c-df23b76f32d8-cert\") pod \"ingress-canary-nhdcg\" (UID: \"00b8825f-8f3b-4da7-929c-df23b76f32d8\") " pod="openshift-ingress-canary/ingress-canary-nhdcg"
Jul 29 03:56:28.429255 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:28.429193    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30b69d69-48d9-48a7-ad3c-dad8542537ee-tmp-dir\") pod \"dns-default-9rmhw\" (UID: \"30b69d69-48d9-48a7-ad3c-dad8542537ee\") " pod="openshift-dns/dns-default-9rmhw"
Jul 29 03:56:28.429255 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:28.429226    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/55fa3eb7-80e7-4b52-bdbe-766ae9a5b93f-data-volume\") pod \"insights-runtime-extractor-mf7ck\" (UID: \"55fa3eb7-80e7-4b52-bdbe-766ae9a5b93f\") " pod="openshift-insights/insights-runtime-extractor-mf7ck"
Jul 29 03:56:28.429255 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:28.429246    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/55fa3eb7-80e7-4b52-bdbe-766ae9a5b93f-crio-socket\") pod \"insights-runtime-extractor-mf7ck\" (UID: \"55fa3eb7-80e7-4b52-bdbe-766ae9a5b93f\") " pod="openshift-insights/insights-runtime-extractor-mf7ck"
Jul 29 03:56:28.429479 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:28.429292    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54wbl\" (UniqueName: \"kubernetes.io/projected/30b69d69-48d9-48a7-ad3c-dad8542537ee-kube-api-access-54wbl\") pod \"dns-default-9rmhw\" (UID: \"30b69d69-48d9-48a7-ad3c-dad8542537ee\") " pod="openshift-dns/dns-default-9rmhw"
Jul 29 03:56:28.429479 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:28.429320    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/55fa3eb7-80e7-4b52-bdbe-766ae9a5b93f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mf7ck\" (UID: \"55fa3eb7-80e7-4b52-bdbe-766ae9a5b93f\") " pod="openshift-insights/insights-runtime-extractor-mf7ck"
Jul 29 03:56:28.429479 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:28.429341    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/30b69d69-48d9-48a7-ad3c-dad8542537ee-metrics-tls\") pod \"dns-default-9rmhw\" (UID: \"30b69d69-48d9-48a7-ad3c-dad8542537ee\") " pod="openshift-dns/dns-default-9rmhw"
Jul 29 03:56:28.429479 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:28.429379    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nzbq\" (UniqueName: \"kubernetes.io/projected/00b8825f-8f3b-4da7-929c-df23b76f32d8-kube-api-access-6nzbq\") pod \"ingress-canary-nhdcg\" (UID: \"00b8825f-8f3b-4da7-929c-df23b76f32d8\") " pod="openshift-ingress-canary/ingress-canary-nhdcg"
Jul 29 03:56:28.429479 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:28.429424    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/55fa3eb7-80e7-4b52-bdbe-766ae9a5b93f-crio-socket\") pod \"insights-runtime-extractor-mf7ck\" (UID: \"55fa3eb7-80e7-4b52-bdbe-766ae9a5b93f\") " pod="openshift-insights/insights-runtime-extractor-mf7ck"
Jul 29 03:56:28.429479 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:28.429449    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/30b69d69-48d9-48a7-ad3c-dad8542537ee-config-volume\") pod \"dns-default-9rmhw\" (UID: \"30b69d69-48d9-48a7-ad3c-dad8542537ee\") " pod="openshift-dns/dns-default-9rmhw"
Jul 29 03:56:28.429755 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:28.429576    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmls4\" (UniqueName: \"kubernetes.io/projected/55fa3eb7-80e7-4b52-bdbe-766ae9a5b93f-kube-api-access-mmls4\") pod \"insights-runtime-extractor-mf7ck\" (UID: \"55fa3eb7-80e7-4b52-bdbe-766ae9a5b93f\") " pod="openshift-insights/insights-runtime-extractor-mf7ck"
Jul 29 03:56:28.429755 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:28.429602    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/55fa3eb7-80e7-4b52-bdbe-766ae9a5b93f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mf7ck\" (UID: \"55fa3eb7-80e7-4b52-bdbe-766ae9a5b93f\") " pod="openshift-insights/insights-runtime-extractor-mf7ck"
Jul 29 03:56:28.429755 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:28.429607    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/55fa3eb7-80e7-4b52-bdbe-766ae9a5b93f-data-volume\") pod \"insights-runtime-extractor-mf7ck\" (UID: \"55fa3eb7-80e7-4b52-bdbe-766ae9a5b93f\") " pod="openshift-insights/insights-runtime-extractor-mf7ck"
Jul 29 03:56:28.429755 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:28.429642    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/00b8825f-8f3b-4da7-929c-df23b76f32d8-cert\") pod \"ingress-canary-nhdcg\" (UID: \"00b8825f-8f3b-4da7-929c-df23b76f32d8\") " pod="openshift-ingress-canary/ingress-canary-nhdcg"
Jul 29 03:56:28.429959 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:28.429892    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/55fa3eb7-80e7-4b52-bdbe-766ae9a5b93f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mf7ck\" (UID: \"55fa3eb7-80e7-4b52-bdbe-766ae9a5b93f\") " pod="openshift-insights/insights-runtime-extractor-mf7ck"
Jul 29 03:56:28.433634 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:28.433610    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/55fa3eb7-80e7-4b52-bdbe-766ae9a5b93f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mf7ck\" (UID: \"55fa3eb7-80e7-4b52-bdbe-766ae9a5b93f\") " pod="openshift-insights/insights-runtime-extractor-mf7ck"
Jul 29 03:56:28.433708 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:28.433653    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/00b8825f-8f3b-4da7-929c-df23b76f32d8-cert\") pod \"ingress-canary-nhdcg\" (UID: \"00b8825f-8f3b-4da7-929c-df23b76f32d8\") " pod="openshift-ingress-canary/ingress-canary-nhdcg"
Jul 29 03:56:28.437753 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:28.437734    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmls4\" (UniqueName: \"kubernetes.io/projected/55fa3eb7-80e7-4b52-bdbe-766ae9a5b93f-kube-api-access-mmls4\") pod \"insights-runtime-extractor-mf7ck\" (UID: \"55fa3eb7-80e7-4b52-bdbe-766ae9a5b93f\") " pod="openshift-insights/insights-runtime-extractor-mf7ck"
Jul 29 03:56:28.437877 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:28.437780    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nzbq\" (UniqueName: \"kubernetes.io/projected/00b8825f-8f3b-4da7-929c-df23b76f32d8-kube-api-access-6nzbq\") pod \"ingress-canary-nhdcg\" (UID: \"00b8825f-8f3b-4da7-929c-df23b76f32d8\") " pod="openshift-ingress-canary/ingress-canary-nhdcg"
Jul 29 03:56:28.478877 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:28.478857    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nhdcg"
Jul 29 03:56:28.498012 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:28.497994    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mf7ck"
Jul 29 03:56:28.529974 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:28.529954    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30b69d69-48d9-48a7-ad3c-dad8542537ee-tmp-dir\") pod \"dns-default-9rmhw\" (UID: \"30b69d69-48d9-48a7-ad3c-dad8542537ee\") " pod="openshift-dns/dns-default-9rmhw"
Jul 29 03:56:28.530091 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:28.529989    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54wbl\" (UniqueName: \"kubernetes.io/projected/30b69d69-48d9-48a7-ad3c-dad8542537ee-kube-api-access-54wbl\") pod \"dns-default-9rmhw\" (UID: \"30b69d69-48d9-48a7-ad3c-dad8542537ee\") " pod="openshift-dns/dns-default-9rmhw"
Jul 29 03:56:28.530091 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:28.530029    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/30b69d69-48d9-48a7-ad3c-dad8542537ee-metrics-tls\") pod \"dns-default-9rmhw\" (UID: \"30b69d69-48d9-48a7-ad3c-dad8542537ee\") " pod="openshift-dns/dns-default-9rmhw"
Jul 29 03:56:28.530091 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:28.530048    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/30b69d69-48d9-48a7-ad3c-dad8542537ee-config-volume\") pod \"dns-default-9rmhw\" (UID: \"30b69d69-48d9-48a7-ad3c-dad8542537ee\") " pod="openshift-dns/dns-default-9rmhw"
Jul 29 03:56:28.530318 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:28.530298    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30b69d69-48d9-48a7-ad3c-dad8542537ee-tmp-dir\") pod \"dns-default-9rmhw\" (UID: \"30b69d69-48d9-48a7-ad3c-dad8542537ee\") " pod="openshift-dns/dns-default-9rmhw"
Jul 29 03:56:28.530549 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:28.530533    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/30b69d69-48d9-48a7-ad3c-dad8542537ee-config-volume\") pod \"dns-default-9rmhw\" (UID: \"30b69d69-48d9-48a7-ad3c-dad8542537ee\") " pod="openshift-dns/dns-default-9rmhw"
Jul 29 03:56:28.532380 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:28.532359    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/30b69d69-48d9-48a7-ad3c-dad8542537ee-metrics-tls\") pod \"dns-default-9rmhw\" (UID: \"30b69d69-48d9-48a7-ad3c-dad8542537ee\") " pod="openshift-dns/dns-default-9rmhw"
Jul 29 03:56:28.541804 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:28.541781    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54wbl\" (UniqueName: \"kubernetes.io/projected/30b69d69-48d9-48a7-ad3c-dad8542537ee-kube-api-access-54wbl\") pod \"dns-default-9rmhw\" (UID: \"30b69d69-48d9-48a7-ad3c-dad8542537ee\") " pod="openshift-dns/dns-default-9rmhw"
Jul 29 03:56:28.573949 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:28.573915    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9rmhw"
Jul 29 03:56:28.684669 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:28.684605    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nhdcg"]
Jul 29 03:56:28.697605 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:56:28.697580    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00b8825f_8f3b_4da7_929c_df23b76f32d8.slice/crio-bb5bdd94395813e8ee24e484d4a6b1b3eabb1b598ed2ab640fcb537486a17baa WatchSource:0}: Error finding container bb5bdd94395813e8ee24e484d4a6b1b3eabb1b598ed2ab640fcb537486a17baa: Status 404 returned error can't find the container with id bb5bdd94395813e8ee24e484d4a6b1b3eabb1b598ed2ab640fcb537486a17baa
Jul 29 03:56:28.708569 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:28.708543    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mf7ck"]
Jul 29 03:56:28.711518 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:56:28.711488    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55fa3eb7_80e7_4b52_bdbe_766ae9a5b93f.slice/crio-cffaa01906c17f6a6abd5dec951eb6b0d8db21c3ba28a531b18599a2b4a060b7 WatchSource:0}: Error finding container cffaa01906c17f6a6abd5dec951eb6b0d8db21c3ba28a531b18599a2b4a060b7: Status 404 returned error can't find the container with id cffaa01906c17f6a6abd5dec951eb6b0d8db21c3ba28a531b18599a2b4a060b7
Jul 29 03:56:28.748516 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:28.748489    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9rmhw"]
Jul 29 03:56:28.752066 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:56:28.752035    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30b69d69_48d9_48a7_ad3c_dad8542537ee.slice/crio-00824842edaf0785eefedc3bf4f4076f96120538453cde83fd6544fb53ce8ed1 WatchSource:0}: Error finding container 00824842edaf0785eefedc3bf4f4076f96120538453cde83fd6544fb53ce8ed1: Status 404 returned error can't find the container with id 00824842edaf0785eefedc3bf4f4076f96120538453cde83fd6544fb53ce8ed1
Jul 29 03:56:28.829185 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:28.829156    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vvzjp"
Jul 29 03:56:28.832001 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:28.831985    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 29 03:56:29.239903 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:29.239840    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9rmhw" event={"ID":"30b69d69-48d9-48a7-ad3c-dad8542537ee","Type":"ContainerStarted","Data":"00824842edaf0785eefedc3bf4f4076f96120538453cde83fd6544fb53ce8ed1"}
Jul 29 03:56:29.241698 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:29.241606    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mf7ck" event={"ID":"55fa3eb7-80e7-4b52-bdbe-766ae9a5b93f","Type":"ContainerStarted","Data":"cffaa01906c17f6a6abd5dec951eb6b0d8db21c3ba28a531b18599a2b4a060b7"}
Jul 29 03:56:29.243082 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:29.243024    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nhdcg" event={"ID":"00b8825f-8f3b-4da7-929c-df23b76f32d8","Type":"ContainerStarted","Data":"bb5bdd94395813e8ee24e484d4a6b1b3eabb1b598ed2ab640fcb537486a17baa"}
Jul 29 03:56:29.829808 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:29.829005    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ddj8"
Jul 29 03:56:29.830702 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:29.830147    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8222s"
Jul 29 03:56:29.833040 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:29.833005    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 29 03:56:29.833381 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:29.833265    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-whxkd\""
Jul 29 03:56:29.833664 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:29.833645    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 29 03:56:29.833889 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:29.833872    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-h8jgt\""
Jul 29 03:56:29.834110 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:29.834089    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 29 03:56:34.039919 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:34.039885    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-749c48b96f-q7lv6"]
Jul 29 03:56:34.049043 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:34.049019    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-749c48b96f-q7lv6"
Jul 29 03:56:34.052331 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:34.052308    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-749c48b96f-q7lv6"]
Jul 29 03:56:34.053178 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:34.052843    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 29 03:56:34.053178 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:34.053039    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 29 03:56:34.054338 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:34.053492    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 29 03:56:34.054338 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:34.053681    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 29 03:56:34.054338 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:34.053879    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 29 03:56:34.054338 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:34.054067    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 29 03:56:34.054338 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:34.054148    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-shspq\""
Jul 29 03:56:34.054338 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:34.054271    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 29 03:56:34.175696 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:34.174578    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/99a37005-e877-417e-a9a2-98a9bed210f5-service-ca\") pod \"console-749c48b96f-q7lv6\" (UID: \"99a37005-e877-417e-a9a2-98a9bed210f5\") " pod="openshift-console/console-749c48b96f-q7lv6"
Jul 29 03:56:34.175696 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:34.174655    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/99a37005-e877-417e-a9a2-98a9bed210f5-console-serving-cert\") pod \"console-749c48b96f-q7lv6\" (UID: \"99a37005-e877-417e-a9a2-98a9bed210f5\") " pod="openshift-console/console-749c48b96f-q7lv6"
Jul 29 03:56:34.175696 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:34.174696    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/99a37005-e877-417e-a9a2-98a9bed210f5-console-oauth-config\") pod \"console-749c48b96f-q7lv6\" (UID: \"99a37005-e877-417e-a9a2-98a9bed210f5\") " pod="openshift-console/console-749c48b96f-q7lv6"
Jul 29 03:56:34.175696 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:34.174733    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/99a37005-e877-417e-a9a2-98a9bed210f5-oauth-serving-cert\") pod \"console-749c48b96f-q7lv6\" (UID: \"99a37005-e877-417e-a9a2-98a9bed210f5\") " pod="openshift-console/console-749c48b96f-q7lv6"
Jul 29 03:56:34.175696 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:34.174769    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/99a37005-e877-417e-a9a2-98a9bed210f5-console-config\") pod \"console-749c48b96f-q7lv6\" (UID: \"99a37005-e877-417e-a9a2-98a9bed210f5\") " pod="openshift-console/console-749c48b96f-q7lv6"
Jul 29 03:56:34.175696 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:34.174800    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8flp\" (UniqueName: \"kubernetes.io/projected/99a37005-e877-417e-a9a2-98a9bed210f5-kube-api-access-g8flp\") pod \"console-749c48b96f-q7lv6\" (UID: \"99a37005-e877-417e-a9a2-98a9bed210f5\") " pod="openshift-console/console-749c48b96f-q7lv6"
Jul 29 03:56:34.258610 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:34.258574    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9rmhw" event={"ID":"30b69d69-48d9-48a7-ad3c-dad8542537ee","Type":"ContainerStarted","Data":"d2aa199d509574a6e7a93474abe32af86355898c56f4229afb13423a96e6bdff"}
Jul 29 03:56:34.260166 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:34.260140    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nhdcg" event={"ID":"00b8825f-8f3b-4da7-929c-df23b76f32d8","Type":"ContainerStarted","Data":"4dcd75622fb7739c98e5eeb17ed9db40eb160c756094302ccffd4b9c5e140452"}
Jul 29 03:56:34.275538 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:34.275503    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/99a37005-e877-417e-a9a2-98a9bed210f5-console-oauth-config\") pod \"console-749c48b96f-q7lv6\" (UID: \"99a37005-e877-417e-a9a2-98a9bed210f5\") " pod="openshift-console/console-749c48b96f-q7lv6"
Jul 29 03:56:34.275684 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:34.275556    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/99a37005-e877-417e-a9a2-98a9bed210f5-oauth-serving-cert\") pod \"console-749c48b96f-q7lv6\" (UID: \"99a37005-e877-417e-a9a2-98a9bed210f5\") " pod="openshift-console/console-749c48b96f-q7lv6"
Jul 29 03:56:34.275684 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:34.275586    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/99a37005-e877-417e-a9a2-98a9bed210f5-console-config\") pod \"console-749c48b96f-q7lv6\" (UID: \"99a37005-e877-417e-a9a2-98a9bed210f5\") " pod="openshift-console/console-749c48b96f-q7lv6"
Jul 29 03:56:34.275684 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:34.275615    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8flp\" (UniqueName: \"kubernetes.io/projected/99a37005-e877-417e-a9a2-98a9bed210f5-kube-api-access-g8flp\") pod \"console-749c48b96f-q7lv6\" (UID: \"99a37005-e877-417e-a9a2-98a9bed210f5\") " pod="openshift-console/console-749c48b96f-q7lv6"
Jul 29 03:56:34.275684 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:34.275663    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/99a37005-e877-417e-a9a2-98a9bed210f5-service-ca\") pod \"console-749c48b96f-q7lv6\" (UID: \"99a37005-e877-417e-a9a2-98a9bed210f5\") " pod="openshift-console/console-749c48b96f-q7lv6"
Jul 29 03:56:34.275930 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:34.275717    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/99a37005-e877-417e-a9a2-98a9bed210f5-console-serving-cert\") pod \"console-749c48b96f-q7lv6\" (UID: \"99a37005-e877-417e-a9a2-98a9bed210f5\") " pod="openshift-console/console-749c48b96f-q7lv6"
Jul 29 03:56:34.277281 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:34.276766    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/99a37005-e877-417e-a9a2-98a9bed210f5-oauth-serving-cert\") pod \"console-749c48b96f-q7lv6\" (UID: \"99a37005-e877-417e-a9a2-98a9bed210f5\") " pod="openshift-console/console-749c48b96f-q7lv6"
Jul 29 03:56:34.277281 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:34.277241    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/99a37005-e877-417e-a9a2-98a9bed210f5-service-ca\") pod \"console-749c48b96f-q7lv6\" (UID: \"99a37005-e877-417e-a9a2-98a9bed210f5\") " pod="openshift-console/console-749c48b96f-q7lv6"
Jul 29 03:56:34.277876 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:34.277853    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/99a37005-e877-417e-a9a2-98a9bed210f5-console-config\") pod \"console-749c48b96f-q7lv6\" (UID: \"99a37005-e877-417e-a9a2-98a9bed210f5\") " pod="openshift-console/console-749c48b96f-q7lv6"
Jul 29 03:56:34.278774 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:34.278751    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/99a37005-e877-417e-a9a2-98a9bed210f5-console-oauth-config\") pod \"console-749c48b96f-q7lv6\" (UID: \"99a37005-e877-417e-a9a2-98a9bed210f5\") " pod="openshift-console/console-749c48b96f-q7lv6"
Jul 29 03:56:34.279113 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:34.279095    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/99a37005-e877-417e-a9a2-98a9bed210f5-console-serving-cert\") pod \"console-749c48b96f-q7lv6\" (UID: \"99a37005-e877-417e-a9a2-98a9bed210f5\") " pod="openshift-console/console-749c48b96f-q7lv6"
Jul 29 03:56:34.289284 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:34.289257    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8flp\" (UniqueName: \"kubernetes.io/projected/99a37005-e877-417e-a9a2-98a9bed210f5-kube-api-access-g8flp\") pod \"console-749c48b96f-q7lv6\" (UID: \"99a37005-e877-417e-a9a2-98a9bed210f5\") " pod="openshift-console/console-749c48b96f-q7lv6"
Jul 29 03:56:34.360254 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:34.360231    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-749c48b96f-q7lv6"
Jul 29 03:56:34.505761 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:34.505703    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-nhdcg" podStartSLOduration=1.0723376039999999 podStartE2EDuration="6.505677281s" totalImagesPullingTime="5.433339677s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 03:56:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:56:34.276071093 +0000 UTC m=+59.006202184" watchObservedRunningTime="2026-07-29 03:56:34.505677281 +0000 UTC m=+59.235808350"
Jul 29 03:56:34.506855 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:34.506834    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-749c48b96f-q7lv6"]
Jul 29 03:56:34.509330 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:56:34.509306    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99a37005_e877_417e_a9a2_98a9bed210f5.slice/crio-a90de46001955987f51c2d468c40caec191cf27d06d8c158440b2b29ab4d084c WatchSource:0}: Error finding container a90de46001955987f51c2d468c40caec191cf27d06d8c158440b2b29ab4d084c: Status 404 returned error can't find the container with id a90de46001955987f51c2d468c40caec191cf27d06d8c158440b2b29ab4d084c
Jul 29 03:56:35.266297 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:35.266222    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9rmhw" event={"ID":"30b69d69-48d9-48a7-ad3c-dad8542537ee","Type":"ContainerStarted","Data":"05bdaae690fb3c9a2a1e510c7569d0e0d0def8157f13cd7bd0e13fefa20ac8fe"}
Jul 29 03:56:35.266757 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:35.266378    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-9rmhw"
Jul 29 03:56:35.268299 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:35.268267    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-749c48b96f-q7lv6" event={"ID":"99a37005-e877-417e-a9a2-98a9bed210f5","Type":"ContainerStarted","Data":"a90de46001955987f51c2d468c40caec191cf27d06d8c158440b2b29ab4d084c"}
Jul 29 03:56:35.272007 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:35.271650    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mf7ck" event={"ID":"55fa3eb7-80e7-4b52-bdbe-766ae9a5b93f","Type":"ContainerStarted","Data":"377b7e222e0b28f9607539a6aa2ad4c331e19150b975c76b4aa281604b62da4a"}
Jul 29 03:56:35.288755 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:35.288710    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-9rmhw" podStartSLOduration=1.91100479 podStartE2EDuration="7.288694891s" totalImagesPullingTime="5.377690101s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 03:56:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:56:35.287626917 +0000 UTC m=+60.017757987" watchObservedRunningTime="2026-07-29 03:56:35.288694891 +0000 UTC m=+60.018825951"
Jul 29 03:56:35.448469 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:35.448437    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5477ff644-xzdff"]
Jul 29 03:56:35.461770 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:35.461736    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5477ff644-xzdff"]
Jul 29 03:56:35.461939 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:35.461873    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5477ff644-xzdff"
Jul 29 03:56:35.488421 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:35.487843    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 29 03:56:35.588363 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:35.587990    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3c5a62a3-4ad6-4fbb-a360-7a29be811e12-console-serving-cert\") pod \"console-5477ff644-xzdff\" (UID: \"3c5a62a3-4ad6-4fbb-a360-7a29be811e12\") " pod="openshift-console/console-5477ff644-xzdff"
Jul 29 03:56:35.588363 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:35.588042    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3c5a62a3-4ad6-4fbb-a360-7a29be811e12-console-oauth-config\") pod \"console-5477ff644-xzdff\" (UID: \"3c5a62a3-4ad6-4fbb-a360-7a29be811e12\") " pod="openshift-console/console-5477ff644-xzdff"
Jul 29 03:56:35.588363 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:35.588072    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3c5a62a3-4ad6-4fbb-a360-7a29be811e12-oauth-serving-cert\") pod \"console-5477ff644-xzdff\" (UID: \"3c5a62a3-4ad6-4fbb-a360-7a29be811e12\") " pod="openshift-console/console-5477ff644-xzdff"
Jul 29 03:56:35.588363 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:35.588106    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3c5a62a3-4ad6-4fbb-a360-7a29be811e12-console-config\") pod \"console-5477ff644-xzdff\" (UID: \"3c5a62a3-4ad6-4fbb-a360-7a29be811e12\") " pod="openshift-console/console-5477ff644-xzdff"
Jul 29 03:56:35.588363 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:35.588146    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s98z\" (UniqueName: \"kubernetes.io/projected/3c5a62a3-4ad6-4fbb-a360-7a29be811e12-kube-api-access-4s98z\") pod \"console-5477ff644-xzdff\" (UID: \"3c5a62a3-4ad6-4fbb-a360-7a29be811e12\") " pod="openshift-console/console-5477ff644-xzdff"
Jul 29 03:56:35.588363 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:35.588187    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c5a62a3-4ad6-4fbb-a360-7a29be811e12-trusted-ca-bundle\") pod \"console-5477ff644-xzdff\" (UID: \"3c5a62a3-4ad6-4fbb-a360-7a29be811e12\") " pod="openshift-console/console-5477ff644-xzdff"
Jul 29 03:56:35.588363 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:35.588216    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3c5a62a3-4ad6-4fbb-a360-7a29be811e12-service-ca\") pod \"console-5477ff644-xzdff\" (UID: \"3c5a62a3-4ad6-4fbb-a360-7a29be811e12\") " pod="openshift-console/console-5477ff644-xzdff"
Jul 29 03:56:35.690761 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:35.688727    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3c5a62a3-4ad6-4fbb-a360-7a29be811e12-console-serving-cert\") pod \"console-5477ff644-xzdff\" (UID: \"3c5a62a3-4ad6-4fbb-a360-7a29be811e12\") " pod="openshift-console/console-5477ff644-xzdff"
Jul 29 03:56:35.690761 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:35.688768    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3c5a62a3-4ad6-4fbb-a360-7a29be811e12-console-oauth-config\") pod \"console-5477ff644-xzdff\" (UID: \"3c5a62a3-4ad6-4fbb-a360-7a29be811e12\") " pod="openshift-console/console-5477ff644-xzdff"
Jul 29 03:56:35.690761 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:35.688797    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3c5a62a3-4ad6-4fbb-a360-7a29be811e12-oauth-serving-cert\") pod \"console-5477ff644-xzdff\" (UID: \"3c5a62a3-4ad6-4fbb-a360-7a29be811e12\") " pod="openshift-console/console-5477ff644-xzdff"
Jul 29 03:56:35.690761 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:35.688862    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3c5a62a3-4ad6-4fbb-a360-7a29be811e12-console-config\") pod \"console-5477ff644-xzdff\" (UID: \"3c5a62a3-4ad6-4fbb-a360-7a29be811e12\") " pod="openshift-console/console-5477ff644-xzdff"
Jul 29 03:56:35.690761 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:35.688902    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s98z\" (UniqueName: \"kubernetes.io/projected/3c5a62a3-4ad6-4fbb-a360-7a29be811e12-kube-api-access-4s98z\") pod \"console-5477ff644-xzdff\" (UID: \"3c5a62a3-4ad6-4fbb-a360-7a29be811e12\") " pod="openshift-console/console-5477ff644-xzdff"
Jul 29 03:56:35.690761 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:35.688938    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c5a62a3-4ad6-4fbb-a360-7a29be811e12-trusted-ca-bundle\") pod \"console-5477ff644-xzdff\" (UID: \"3c5a62a3-4ad6-4fbb-a360-7a29be811e12\") " pod="openshift-console/console-5477ff644-xzdff"
Jul 29 03:56:35.690761 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:35.688970    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3c5a62a3-4ad6-4fbb-a360-7a29be811e12-service-ca\") pod \"console-5477ff644-xzdff\" (UID: \"3c5a62a3-4ad6-4fbb-a360-7a29be811e12\") " pod="openshift-console/console-5477ff644-xzdff"
Jul 29 03:56:35.690761 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:35.689645    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3c5a62a3-4ad6-4fbb-a360-7a29be811e12-service-ca\") pod \"console-5477ff644-xzdff\" (UID: \"3c5a62a3-4ad6-4fbb-a360-7a29be811e12\") " pod="openshift-console/console-5477ff644-xzdff"
Jul 29 03:56:35.690761 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:35.689702    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3c5a62a3-4ad6-4fbb-a360-7a29be811e12-console-config\") pod \"console-5477ff644-xzdff\" (UID: \"3c5a62a3-4ad6-4fbb-a360-7a29be811e12\") " pod="openshift-console/console-5477ff644-xzdff"
Jul 29 03:56:35.690761 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:35.690669    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3c5a62a3-4ad6-4fbb-a360-7a29be811e12-oauth-serving-cert\") pod \"console-5477ff644-xzdff\" (UID: \"3c5a62a3-4ad6-4fbb-a360-7a29be811e12\") " pod="openshift-console/console-5477ff644-xzdff"
Jul 29 03:56:35.691798 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:35.691753    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3c5a62a3-4ad6-4fbb-a360-7a29be811e12-console-serving-cert\") pod \"console-5477ff644-xzdff\" (UID: \"3c5a62a3-4ad6-4fbb-a360-7a29be811e12\") " pod="openshift-console/console-5477ff644-xzdff"
Jul 29 03:56:35.694023 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:35.693883    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3c5a62a3-4ad6-4fbb-a360-7a29be811e12-console-oauth-config\") pod \"console-5477ff644-xzdff\" (UID: \"3c5a62a3-4ad6-4fbb-a360-7a29be811e12\") " pod="openshift-console/console-5477ff644-xzdff"
Jul 29 03:56:35.700622 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:35.700594    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 29 03:56:35.707335 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:35.707302    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s98z\" (UniqueName: \"kubernetes.io/projected/3c5a62a3-4ad6-4fbb-a360-7a29be811e12-kube-api-access-4s98z\") pod \"console-5477ff644-xzdff\" (UID: \"3c5a62a3-4ad6-4fbb-a360-7a29be811e12\") " pod="openshift-console/console-5477ff644-xzdff"
Jul 29 03:56:35.710812 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:35.710772    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c5a62a3-4ad6-4fbb-a360-7a29be811e12-trusted-ca-bundle\") pod \"console-5477ff644-xzdff\" (UID: \"3c5a62a3-4ad6-4fbb-a360-7a29be811e12\") " pod="openshift-console/console-5477ff644-xzdff"
Jul 29 03:56:35.790794 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:35.790768    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5477ff644-xzdff"
Jul 29 03:56:35.970647 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:35.970569    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5477ff644-xzdff"]
Jul 29 03:56:36.189347 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:36.189270    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pznfg"
Jul 29 03:56:36.277058 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:36.277021    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5477ff644-xzdff" event={"ID":"3c5a62a3-4ad6-4fbb-a360-7a29be811e12","Type":"ContainerStarted","Data":"09326713463bbdc2aabe6d8eb4c9f539850d3969140f7ffb1127c90c1305fbf2"}
Jul 29 03:56:39.086067 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:39.086036    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-bm98m"]
Jul 29 03:56:39.092589 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:39.092563    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-bm98m"
Jul 29 03:56:39.095740 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:39.095720    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 29 03:56:39.095993 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:39.095972    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-gkh7s\""
Jul 29 03:56:39.099308 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:39.099286    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-bm98m"]
Jul 29 03:56:39.218305 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:39.216811    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d130e173-40bc-4422-aeae-efe66f929eaa-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-bm98m\" (UID: \"d130e173-40bc-4422-aeae-efe66f929eaa\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-bm98m"
Jul 29 03:56:39.318328 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:39.318300    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d130e173-40bc-4422-aeae-efe66f929eaa-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-bm98m\" (UID: \"d130e173-40bc-4422-aeae-efe66f929eaa\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-bm98m"
Jul 29 03:56:39.320862 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:39.320832    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d130e173-40bc-4422-aeae-efe66f929eaa-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-bm98m\" (UID: \"d130e173-40bc-4422-aeae-efe66f929eaa\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-bm98m"
Jul 29 03:56:39.403788 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:39.403766    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-bm98m"
Jul 29 03:56:39.543569 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:39.543542    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-bm98m"]
Jul 29 03:56:39.546217 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:56:39.546189    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd130e173_40bc_4422_aeae_efe66f929eaa.slice/crio-26f4441c6a83b54e7ac400140b7a3fa1838759630691d1bb3431cab01c03d4ea WatchSource:0}: Error finding container 26f4441c6a83b54e7ac400140b7a3fa1838759630691d1bb3431cab01c03d4ea: Status 404 returned error can't find the container with id 26f4441c6a83b54e7ac400140b7a3fa1838759630691d1bb3431cab01c03d4ea
Jul 29 03:56:40.291375 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:40.291336    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-749c48b96f-q7lv6" event={"ID":"99a37005-e877-417e-a9a2-98a9bed210f5","Type":"ContainerStarted","Data":"a48568c70da6380b9d300f45d2e5d09b1b3ce1f12e4462869f85d1ae99302940"}
Jul 29 03:56:40.293518 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:40.293479    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mf7ck" event={"ID":"55fa3eb7-80e7-4b52-bdbe-766ae9a5b93f","Type":"ContainerStarted","Data":"8a9146557cf58846c1618760962ba79fd87e860e14a62a9f8a3b066f21b6ff49"}
Jul 29 03:56:40.294709 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:40.294681    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-bm98m" event={"ID":"d130e173-40bc-4422-aeae-efe66f929eaa","Type":"ContainerStarted","Data":"26f4441c6a83b54e7ac400140b7a3fa1838759630691d1bb3431cab01c03d4ea"}
Jul 29 03:56:40.296612 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:40.296586    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5477ff644-xzdff" event={"ID":"3c5a62a3-4ad6-4fbb-a360-7a29be811e12","Type":"ContainerStarted","Data":"49d408104cd8475a586318d960ca9b69abd7c472bc9d0a54ea34278cfce53a82"}
Jul 29 03:56:40.333591 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:40.332040    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-749c48b96f-q7lv6" podStartSLOduration=1.496442499 podStartE2EDuration="6.33202025s" totalImagesPullingTime="4.835577751s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 03:56:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:56:40.312496857 +0000 UTC m=+65.042627953" watchObservedRunningTime="2026-07-29 03:56:40.33202025 +0000 UTC m=+65.062151321"
Jul 29 03:56:40.333591 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:40.332205    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5477ff644-xzdff" podStartSLOduration=1.949554861 podStartE2EDuration="5.332198256s" totalImagesPullingTime="3.382643395s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 03:56:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:56:40.331063176 +0000 UTC m=+65.061194262" watchObservedRunningTime="2026-07-29 03:56:40.332198256 +0000 UTC m=+65.062329325"
Jul 29 03:56:40.527767 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:40.527733    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d9e67808-ec21-42a2-bbaf-a28164326ccc-original-pull-secret\") pod \"global-pull-secret-syncer-vvzjp\" (UID: \"d9e67808-ec21-42a2-bbaf-a28164326ccc\") " pod="kube-system/global-pull-secret-syncer-vvzjp"
Jul 29 03:56:40.528075 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:40.527853    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64f62499-e7b7-4249-a1e4-636144783b55-metrics-certs\") pod \"network-metrics-daemon-5ddj8\" (UID: \"64f62499-e7b7-4249-a1e4-636144783b55\") " pod="openshift-multus/network-metrics-daemon-5ddj8"
Jul 29 03:56:40.530689 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:40.530553    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 29 03:56:40.530689 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:40.530580    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 29 03:56:40.541776 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:40.541712    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d9e67808-ec21-42a2-bbaf-a28164326ccc-original-pull-secret\") pod \"global-pull-secret-syncer-vvzjp\" (UID: \"d9e67808-ec21-42a2-bbaf-a28164326ccc\") " pod="kube-system/global-pull-secret-syncer-vvzjp"
Jul 29 03:56:40.543371 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:40.543352    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64f62499-e7b7-4249-a1e4-636144783b55-metrics-certs\") pod \"network-metrics-daemon-5ddj8\" (UID: \"64f62499-e7b7-4249-a1e4-636144783b55\") " pod="openshift-multus/network-metrics-daemon-5ddj8"
Jul 29 03:56:40.629150 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:40.629114    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvmpk\" (UniqueName: \"kubernetes.io/projected/a0ccb95e-5e5c-4a10-9f35-6510b59e84aa-kube-api-access-hvmpk\") pod \"network-check-target-8222s\" (UID: \"a0ccb95e-5e5c-4a10-9f35-6510b59e84aa\") " pod="openshift-network-diagnostics/network-check-target-8222s"
Jul 29 03:56:40.631794 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:40.631768    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 29 03:56:40.642532 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:40.642507    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 29 03:56:40.647154 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:40.646995    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-whxkd\""
Jul 29 03:56:40.653694 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:40.653676    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvmpk\" (UniqueName: \"kubernetes.io/projected/a0ccb95e-5e5c-4a10-9f35-6510b59e84aa-kube-api-access-hvmpk\") pod \"network-check-target-8222s\" (UID: \"a0ccb95e-5e5c-4a10-9f35-6510b59e84aa\") " pod="openshift-network-diagnostics/network-check-target-8222s"
Jul 29 03:56:40.655405 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:40.655382    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ddj8"
Jul 29 03:56:40.796174 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:40.796084    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5ddj8"]
Jul 29 03:56:40.800663 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:56:40.800631    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64f62499_e7b7_4249_a1e4_636144783b55.slice/crio-b9f1c04a38f0f1fe981e574b1d6782f3f3648cf8629ef1abdd6e0dddc4b3c10b WatchSource:0}: Error finding container b9f1c04a38f0f1fe981e574b1d6782f3f3648cf8629ef1abdd6e0dddc4b3c10b: Status 404 returned error can't find the container with id b9f1c04a38f0f1fe981e574b1d6782f3f3648cf8629ef1abdd6e0dddc4b3c10b
Jul 29 03:56:40.837456 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:40.837431    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vvzjp"
Jul 29 03:56:40.955158 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:40.955131    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-h8jgt\""
Jul 29 03:56:40.962609 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:40.962584    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8222s"
Jul 29 03:56:40.974148 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:40.974075    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vvzjp"]
Jul 29 03:56:40.978944 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:56:40.978917    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9e67808_ec21_42a2_bbaf_a28164326ccc.slice/crio-0942889e3325b4dc5ff6562f2306e36318efd53f33966a3f8eff205d16a5fa64 WatchSource:0}: Error finding container 0942889e3325b4dc5ff6562f2306e36318efd53f33966a3f8eff205d16a5fa64: Status 404 returned error can't find the container with id 0942889e3325b4dc5ff6562f2306e36318efd53f33966a3f8eff205d16a5fa64
Jul 29 03:56:41.099446 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:41.099415    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8222s"]
Jul 29 03:56:41.102529 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:56:41.102500    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0ccb95e_5e5c_4a10_9f35_6510b59e84aa.slice/crio-b8f97727e2df9bae43b58a7428faab9493a73188919816705667df735483dd48 WatchSource:0}: Error finding container b8f97727e2df9bae43b58a7428faab9493a73188919816705667df735483dd48: Status 404 returned error can't find the container with id b8f97727e2df9bae43b58a7428faab9493a73188919816705667df735483dd48
Jul 29 03:56:41.302786 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:41.302705    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-bm98m" event={"ID":"d130e173-40bc-4422-aeae-efe66f929eaa","Type":"ContainerStarted","Data":"7107d2a6172c5cebf2f810a74368c26aa28f56883fdaf555be02890db6a1d007"}
Jul 29 03:56:41.303304 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:41.303120    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-bm98m"
Jul 29 03:56:41.304921 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:41.304843    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8222s" event={"ID":"a0ccb95e-5e5c-4a10-9f35-6510b59e84aa","Type":"ContainerStarted","Data":"b8f97727e2df9bae43b58a7428faab9493a73188919816705667df735483dd48"}
Jul 29 03:56:41.306396 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:41.306352    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vvzjp" event={"ID":"d9e67808-ec21-42a2-bbaf-a28164326ccc","Type":"ContainerStarted","Data":"0942889e3325b4dc5ff6562f2306e36318efd53f33966a3f8eff205d16a5fa64"}
Jul 29 03:56:41.308729 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:41.308707    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-bm98m"
Jul 29 03:56:41.309568 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:41.309520    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5ddj8" event={"ID":"64f62499-e7b7-4249-a1e4-636144783b55","Type":"ContainerStarted","Data":"b9f1c04a38f0f1fe981e574b1d6782f3f3648cf8629ef1abdd6e0dddc4b3c10b"}
Jul 29 03:56:41.319030 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:41.318977    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-bm98m" podStartSLOduration=1.183037511 podStartE2EDuration="2.31895994s" totalImagesPullingTime="1.135922429s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 03:56:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:56:41.318132567 +0000 UTC m=+66.048263638" watchObservedRunningTime="2026-07-29 03:56:41.31895994 +0000 UTC m=+66.049091011"
Jul 29 03:56:42.139669 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:42.139635    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-b84ft"]
Jul 29 03:56:42.234352 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:42.234312    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-b84ft"]
Jul 29 03:56:42.234522 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:42.234470    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-b84ft"
Jul 29 03:56:42.237900 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:42.237870    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 29 03:56:42.238028 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:42.237924    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 29 03:56:42.238028 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:42.238019    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 29 03:56:42.239063 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:42.239037    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 29 03:56:42.239265 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:42.239251    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-pntwl\""
Jul 29 03:56:42.239406 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:42.239381    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 29 03:56:42.341393 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:42.341361    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3949548e-4b2f-4982-a946-e383513165bd-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-b84ft\" (UID: \"3949548e-4b2f-4982-a946-e383513165bd\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-b84ft"
Jul 29 03:56:42.341779 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:42.341413    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3949548e-4b2f-4982-a946-e383513165bd-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-b84ft\" (UID: \"3949548e-4b2f-4982-a946-e383513165bd\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-b84ft"
Jul 29 03:56:42.341779 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:42.341475    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3949548e-4b2f-4982-a946-e383513165bd-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-b84ft\" (UID: \"3949548e-4b2f-4982-a946-e383513165bd\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-b84ft"
Jul 29 03:56:42.341779 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:42.341522    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc9w4\" (UniqueName: \"kubernetes.io/projected/3949548e-4b2f-4982-a946-e383513165bd-kube-api-access-hc9w4\") pod \"prometheus-operator-84d69f794d-b84ft\" (UID: \"3949548e-4b2f-4982-a946-e383513165bd\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-b84ft"
Jul 29 03:56:42.442333 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:42.442241    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3949548e-4b2f-4982-a946-e383513165bd-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-b84ft\" (UID: \"3949548e-4b2f-4982-a946-e383513165bd\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-b84ft"
Jul 29 03:56:42.442333 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:42.442311    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3949548e-4b2f-4982-a946-e383513165bd-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-b84ft\" (UID: \"3949548e-4b2f-4982-a946-e383513165bd\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-b84ft"
Jul 29 03:56:42.442566 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:42.442341    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3949548e-4b2f-4982-a946-e383513165bd-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-b84ft\" (UID: \"3949548e-4b2f-4982-a946-e383513165bd\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-b84ft"
Jul 29 03:56:42.442566 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:42.442378    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc9w4\" (UniqueName: \"kubernetes.io/projected/3949548e-4b2f-4982-a946-e383513165bd-kube-api-access-hc9w4\") pod \"prometheus-operator-84d69f794d-b84ft\" (UID: \"3949548e-4b2f-4982-a946-e383513165bd\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-b84ft"
Jul 29 03:56:42.443233 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:42.443023    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3949548e-4b2f-4982-a946-e383513165bd-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-b84ft\" (UID: \"3949548e-4b2f-4982-a946-e383513165bd\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-b84ft"
Jul 29 03:56:42.445292 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:42.445251    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3949548e-4b2f-4982-a946-e383513165bd-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-b84ft\" (UID: \"3949548e-4b2f-4982-a946-e383513165bd\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-b84ft"
Jul 29 03:56:42.445408 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:42.445385    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3949548e-4b2f-4982-a946-e383513165bd-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-b84ft\" (UID: \"3949548e-4b2f-4982-a946-e383513165bd\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-b84ft"
Jul 29 03:56:42.451377 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:42.451349    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc9w4\" (UniqueName: \"kubernetes.io/projected/3949548e-4b2f-4982-a946-e383513165bd-kube-api-access-hc9w4\") pod \"prometheus-operator-84d69f794d-b84ft\" (UID: \"3949548e-4b2f-4982-a946-e383513165bd\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-b84ft"
Jul 29 03:56:42.546011 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:42.545984    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-b84ft"
Jul 29 03:56:43.465258 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:43.465018    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-b84ft"]
Jul 29 03:56:44.323834 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:44.323727    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-b84ft" event={"ID":"3949548e-4b2f-4982-a946-e383513165bd","Type":"ContainerStarted","Data":"fb9123b3628788cc53b1f517d1c97a9c30d6fa930e0ade7f5b0ad6198311d8cb"}
Jul 29 03:56:44.328479 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:44.327848    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5ddj8" event={"ID":"64f62499-e7b7-4249-a1e4-636144783b55","Type":"ContainerStarted","Data":"a7c53dc560462892caa6d80e6f5d51545f0449ebb501ff47bd3ba490b956c6f1"}
Jul 29 03:56:44.328479 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:44.327906    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5ddj8" event={"ID":"64f62499-e7b7-4249-a1e4-636144783b55","Type":"ContainerStarted","Data":"b4516c4102751d434e5a44db11e352aad704f247d59551cb97b25c1b0404ddfb"}
Jul 29 03:56:44.331737 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:44.331694    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mf7ck" event={"ID":"55fa3eb7-80e7-4b52-bdbe-766ae9a5b93f","Type":"ContainerStarted","Data":"b582924eabffca38b1317a21c055cb868a03aa44bca87f3732d89e249d2c6840"}
Jul 29 03:56:44.345581 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:44.345527    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-5ddj8" podStartSLOduration=66.829617241 podStartE2EDuration="1m9.345508155s" totalImagesPullingTime="2.515890914s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 03:55:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:56:44.344049853 +0000 UTC m=+69.074180924" watchObservedRunningTime="2026-07-29 03:56:44.345508155 +0000 UTC m=+69.075639228"
Jul 29 03:56:44.360791 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:44.360766    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-749c48b96f-q7lv6"
Jul 29 03:56:44.360955 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:44.360835    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-749c48b96f-q7lv6"
Jul 29 03:56:44.362384 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:44.361896    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-mf7ck" podStartSLOduration=1.894355393 podStartE2EDuration="16.361877984s" totalImagesPullingTime="14.467522591s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 03:56:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:56:44.360687426 +0000 UTC m=+69.090818523" watchObservedRunningTime="2026-07-29 03:56:44.361877984 +0000 UTC m=+69.092009038"
Jul 29 03:56:44.362746 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:44.362710    2570 patch_prober.go:28] interesting pod/console-749c48b96f-q7lv6 container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.134.0.9:8443/health\": dial tcp 10.134.0.9:8443: connect: connection refused" start-of-body=
Jul 29 03:56:44.362864 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:44.362760    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-749c48b96f-q7lv6" podUID="99a37005-e877-417e-a9a2-98a9bed210f5" containerName="console" probeResult="failure" output="Get \"https://10.134.0.9:8443/health\": dial tcp 10.134.0.9:8443: connect: connection refused"
Jul 29 03:56:45.279228 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:45.279193    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-9rmhw"
Jul 29 03:56:45.791886 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:45.791853    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5477ff644-xzdff"
Jul 29 03:56:45.792067 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:45.791918    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5477ff644-xzdff"
Jul 29 03:56:45.792997 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:45.792971    2570 patch_prober.go:28] interesting pod/console-5477ff644-xzdff container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.134.0.10:8443/health\": dial tcp 10.134.0.10:8443: connect: connection refused" start-of-body=
Jul 29 03:56:45.793090 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:45.793009    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-5477ff644-xzdff" podUID="3c5a62a3-4ad6-4fbb-a360-7a29be811e12" containerName="console" probeResult="failure" output="Get \"https://10.134.0.10:8443/health\": dial tcp 10.134.0.10:8443: connect: connection refused"
Jul 29 03:56:48.349957 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:48.349914    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vvzjp" event={"ID":"d9e67808-ec21-42a2-bbaf-a28164326ccc","Type":"ContainerStarted","Data":"539d315dbbb77e28192dda4e87bbf0eabcea20bebf47eae36be5f0dbf0c9f31c"}
Jul 29 03:56:48.351319 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:48.351296    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8222s" event={"ID":"a0ccb95e-5e5c-4a10-9f35-6510b59e84aa","Type":"ContainerStarted","Data":"055bafc7a1048b43dc27792eea9c0a740d1f8a8b0bd1ad01ae58df667bf9cf9a"}
Jul 29 03:56:48.351413 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:48.351372    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8222s"
Jul 29 03:56:48.352938 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:48.352916    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-b84ft" event={"ID":"3949548e-4b2f-4982-a946-e383513165bd","Type":"ContainerStarted","Data":"6514886d2eb11a054c5b6b74eee2ec916517879235ee83d018a94338be865981"}
Jul 29 03:56:48.353039 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:48.352952    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-b84ft" event={"ID":"3949548e-4b2f-4982-a946-e383513165bd","Type":"ContainerStarted","Data":"eb11563595640ae8ca425ef0b5454e722207e172ebdb175968b45728d74bd669"}
Jul 29 03:56:48.364415 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:48.364377    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-vvzjp" podStartSLOduration=66.938589409 podStartE2EDuration="1m13.364366146s" totalImagesPullingTime="6.425776737s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 03:55:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:56:48.363751868 +0000 UTC m=+73.093882939" watchObservedRunningTime="2026-07-29 03:56:48.364366146 +0000 UTC m=+73.094497216"
Jul 29 03:56:48.378916 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:48.378879    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-8222s" podStartSLOduration=67.077069412 podStartE2EDuration="1m13.378868764s" totalImagesPullingTime="6.301799352s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 03:55:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:56:48.378145302 +0000 UTC m=+73.108276389" watchObservedRunningTime="2026-07-29 03:56:48.378868764 +0000 UTC m=+73.108999833"
Jul 29 03:56:48.398335 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:48.398281    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-b84ft" podStartSLOduration=2.474410484 podStartE2EDuration="6.398265481s" totalImagesPullingTime="3.923854997s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 03:56:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:56:48.397286281 +0000 UTC m=+73.127417362" watchObservedRunningTime="2026-07-29 03:56:48.398265481 +0000 UTC m=+73.128396551"
Jul 29 03:56:50.478238 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:50.478206    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-bwtm8"]
Jul 29 03:56:50.480705 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:50.480679    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-bwtm8"
Jul 29 03:56:50.483398 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:50.483377    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 29 03:56:50.483532 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:50.483402    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-nd4tq\""
Jul 29 03:56:50.483532 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:50.483459    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 29 03:56:50.491566 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:50.491532    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-bwtm8"]
Jul 29 03:56:50.499353 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:50.499332    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/364ce126-2396-443f-a420-7be5f74d1494-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-bwtm8\" (UID: \"364ce126-2396-443f-a420-7be5f74d1494\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-bwtm8"
Jul 29 03:56:50.499448 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:50.499360    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hsd2\" (UniqueName: \"kubernetes.io/projected/364ce126-2396-443f-a420-7be5f74d1494-kube-api-access-8hsd2\") pod \"openshift-state-metrics-b8f49f5c6-bwtm8\" (UID: \"364ce126-2396-443f-a420-7be5f74d1494\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-bwtm8"
Jul 29 03:56:50.499448 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:50.499389    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/364ce126-2396-443f-a420-7be5f74d1494-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-bwtm8\" (UID: \"364ce126-2396-443f-a420-7be5f74d1494\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-bwtm8"
Jul 29 03:56:50.499448 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:50.499412    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/364ce126-2396-443f-a420-7be5f74d1494-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-bwtm8\" (UID: \"364ce126-2396-443f-a420-7be5f74d1494\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-bwtm8"
Jul 29 03:56:50.502512 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:50.502494    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-f7hkw"]
Jul 29 03:56:50.504751 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:50.504733    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-f7hkw"
Jul 29 03:56:50.507105 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:50.507084    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 29 03:56:50.507184 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:50.507169    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 29 03:56:50.507991 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:50.507975    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 29 03:56:50.508042 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:50.508013    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-64z8v\""
Jul 29 03:56:50.514754 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:50.514738    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-k78tt"]
Jul 29 03:56:50.517149 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:50.516913    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-k78tt"
Jul 29 03:56:50.518424 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:50.518402    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-f7hkw"]
Jul 29 03:56:50.519451 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:50.519433    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 29 03:56:50.519529 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:50.519468    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 29 03:56:50.519791 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:50.519773    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-b4bvh\""
Jul 29 03:56:50.519905 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:50.519887    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 29 03:56:50.600008 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:50.599983    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/254912df-9a41-48da-834d-7098495e379c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-f7hkw\" (UID: \"254912df-9a41-48da-834d-7098495e379c\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-f7hkw"
Jul 29 03:56:50.600008 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:50.600011    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/354094fb-89f8-4929-aae9-dc99c906d34d-sys\") pod \"node-exporter-k78tt\" (UID: \"354094fb-89f8-4929-aae9-dc99c906d34d\") " pod="openshift-monitoring/node-exporter-k78tt"
Jul 29 03:56:50.600176 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:50.600036    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/354094fb-89f8-4929-aae9-dc99c906d34d-node-exporter-tls\") pod \"node-exporter-k78tt\" (UID: \"354094fb-89f8-4929-aae9-dc99c906d34d\") " pod="openshift-monitoring/node-exporter-k78tt"
Jul 29 03:56:50.600176 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:50.600088    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm5gj\" (UniqueName: \"kubernetes.io/projected/254912df-9a41-48da-834d-7098495e379c-kube-api-access-dm5gj\") pod \"kube-state-metrics-56d9f944b4-f7hkw\" (UID: \"254912df-9a41-48da-834d-7098495e379c\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-f7hkw"
Jul 29 03:56:50.600176 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:50.600119    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/354094fb-89f8-4929-aae9-dc99c906d34d-metrics-client-ca\") pod \"node-exporter-k78tt\" (UID: \"354094fb-89f8-4929-aae9-dc99c906d34d\") " pod="openshift-monitoring/node-exporter-k78tt"
Jul 29 03:56:50.600176 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:50.600144    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/354094fb-89f8-4929-aae9-dc99c906d34d-node-exporter-wtmp\") pod \"node-exporter-k78tt\" (UID: \"354094fb-89f8-4929-aae9-dc99c906d34d\") " pod="openshift-monitoring/node-exporter-k78tt"
Jul 29 03:56:50.600176 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:50.600170    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/364ce126-2396-443f-a420-7be5f74d1494-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-bwtm8\" (UID: \"364ce126-2396-443f-a420-7be5f74d1494\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-bwtm8"
Jul 29 03:56:50.600318 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:50.600191    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d27f2\" (UniqueName: \"kubernetes.io/projected/354094fb-89f8-4929-aae9-dc99c906d34d-kube-api-access-d27f2\") pod \"node-exporter-k78tt\" (UID: \"354094fb-89f8-4929-aae9-dc99c906d34d\") " pod="openshift-monitoring/node-exporter-k78tt"
Jul 29 03:56:50.600318 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:50.600222    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hsd2\" (UniqueName: \"kubernetes.io/projected/364ce126-2396-443f-a420-7be5f74d1494-kube-api-access-8hsd2\") pod \"openshift-state-metrics-b8f49f5c6-bwtm8\" (UID: \"364ce126-2396-443f-a420-7be5f74d1494\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-bwtm8"
Jul 29 03:56:50.600318 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:50.600251    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/254912df-9a41-48da-834d-7098495e379c-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-f7hkw\" (UID: \"254912df-9a41-48da-834d-7098495e379c\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-f7hkw"
Jul 29 03:56:50.600318 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:50.600274    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/254912df-9a41-48da-834d-7098495e379c-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-f7hkw\" (UID: \"254912df-9a41-48da-834d-7098495e379c\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-f7hkw"
Jul 29 03:56:50.600318 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:56:50.600277    2570 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Jul 29 03:56:50.600501 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:56:50.600329    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/364ce126-2396-443f-a420-7be5f74d1494-openshift-state-metrics-tls podName:364ce126-2396-443f-a420-7be5f74d1494 nodeName:}" failed. No retries permitted until 2026-07-29 03:56:51.100311938 +0000 UTC m=+75.830442986 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/364ce126-2396-443f-a420-7be5f74d1494-openshift-state-metrics-tls") pod "openshift-state-metrics-b8f49f5c6-bwtm8" (UID: "364ce126-2396-443f-a420-7be5f74d1494") : secret "openshift-state-metrics-tls" not found
Jul 29 03:56:50.600501 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:50.600330    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/354094fb-89f8-4929-aae9-dc99c906d34d-root\") pod \"node-exporter-k78tt\" (UID: \"354094fb-89f8-4929-aae9-dc99c906d34d\") " pod="openshift-monitoring/node-exporter-k78tt"
Jul 29 03:56:50.600501 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:50.600349    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/354094fb-89f8-4929-aae9-dc99c906d34d-node-exporter-textfile\") pod \"node-exporter-k78tt\" (UID: \"354094fb-89f8-4929-aae9-dc99c906d34d\") " pod="openshift-monitoring/node-exporter-k78tt"
Jul 29 03:56:50.600501 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:50.600371    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/254912df-9a41-48da-834d-7098495e379c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-f7hkw\" (UID: \"254912df-9a41-48da-834d-7098495e379c\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-f7hkw"
Jul 29 03:56:50.600501 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:50.600405    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/364ce126-2396-443f-a420-7be5f74d1494-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-bwtm8\" (UID: \"364ce126-2396-443f-a420-7be5f74d1494\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-bwtm8"
Jul 29 03:56:50.600501 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:50.600426    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/254912df-9a41-48da-834d-7098495e379c-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-f7hkw\" (UID: \"254912df-9a41-48da-834d-7098495e379c\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-f7hkw"
Jul 29 03:56:50.600501 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:50.600446    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/364ce126-2396-443f-a420-7be5f74d1494-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-bwtm8\" (UID: \"364ce126-2396-443f-a420-7be5f74d1494\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-bwtm8"
Jul 29 03:56:50.600501 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:50.600463    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/354094fb-89f8-4929-aae9-dc99c906d34d-node-exporter-accelerators-collector-config\") pod \"node-exporter-k78tt\" (UID: \"354094fb-89f8-4929-aae9-dc99c906d34d\") " pod="openshift-monitoring/node-exporter-k78tt"
Jul 29 03:56:50.600501 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:50.600481    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/354094fb-89f8-4929-aae9-dc99c906d34d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-k78tt\" (UID: \"354094fb-89f8-4929-aae9-dc99c906d34d\") " pod="openshift-monitoring/node-exporter-k78tt"
Jul 29 03:56:50.601092 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:50.601069    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/364ce126-2396-443f-a420-7be5f74d1494-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-bwtm8\" (UID: \"364ce126-2396-443f-a420-7be5f74d1494\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-bwtm8"
Jul 29 03:56:50.602791 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:50.602773    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/364ce126-2396-443f-a420-7be5f74d1494-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-bwtm8\" (UID: \"364ce126-2396-443f-a420-7be5f74d1494\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-bwtm8"
Jul 29 03:56:50.610980 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:50.610961    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hsd2\" (UniqueName: \"kubernetes.io/projected/364ce126-2396-443f-a420-7be5f74d1494-kube-api-access-8hsd2\") pod \"openshift-state-metrics-b8f49f5c6-bwtm8\" (UID: \"364ce126-2396-443f-a420-7be5f74d1494\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-bwtm8"
Jul 29 03:56:50.700943 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:50.700919    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/254912df-9a41-48da-834d-7098495e379c-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-f7hkw\" (UID: \"254912df-9a41-48da-834d-7098495e379c\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-f7hkw"
Jul 29 03:56:50.701066 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:50.700951    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/354094fb-89f8-4929-aae9-dc99c906d34d-node-exporter-accelerators-collector-config\") pod \"node-exporter-k78tt\" (UID: \"354094fb-89f8-4929-aae9-dc99c906d34d\") " pod="openshift-monitoring/node-exporter-k78tt"
Jul 29 03:56:50.701066 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:50.700970    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/354094fb-89f8-4929-aae9-dc99c906d34d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-k78tt\" (UID: \"354094fb-89f8-4929-aae9-dc99c906d34d\") " pod="openshift-monitoring/node-exporter-k78tt"
Jul 29 03:56:50.701066 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:50.700994    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/254912df-9a41-48da-834d-7098495e379c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-f7hkw\" (UID: \"254912df-9a41-48da-834d-7098495e379c\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-f7hkw"
Jul 29 03:56:50.701066 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:50.701016    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/354094fb-89f8-4929-aae9-dc99c906d34d-sys\") pod \"node-exporter-k78tt\" (UID: \"354094fb-89f8-4929-aae9-dc99c906d34d\") " pod="openshift-monitoring/node-exporter-k78tt"
Jul 29 03:56:50.701066 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:50.701050    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/354094fb-89f8-4929-aae9-dc99c906d34d-node-exporter-tls\") pod \"node-exporter-k78tt\" (UID: \"354094fb-89f8-4929-aae9-dc99c906d34d\") " pod="openshift-monitoring/node-exporter-k78tt"
Jul 29 03:56:50.701323 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:50.701085    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm5gj\" (UniqueName: \"kubernetes.io/projected/254912df-9a41-48da-834d-7098495e379c-kube-api-access-dm5gj\") pod \"kube-state-metrics-56d9f944b4-f7hkw\" (UID: \"254912df-9a41-48da-834d-7098495e379c\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-f7hkw"
Jul 29 03:56:50.701323 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:50.701107    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/354094fb-89f8-4929-aae9-dc99c906d34d-metrics-client-ca\") pod \"node-exporter-k78tt\" (UID: \"354094fb-89f8-4929-aae9-dc99c906d34d\") " pod="openshift-monitoring/node-exporter-k78tt"
Jul 29 03:56:50.701323 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:50.701133    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/354094fb-89f8-4929-aae9-dc99c906d34d-node-exporter-wtmp\") pod \"node-exporter-k78tt\" (UID: \"354094fb-89f8-4929-aae9-dc99c906d34d\") " pod="openshift-monitoring/node-exporter-k78tt"
Jul 29 03:56:50.701323 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:50.701176    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d27f2\" (UniqueName: \"kubernetes.io/projected/354094fb-89f8-4929-aae9-dc99c906d34d-kube-api-access-d27f2\") pod \"node-exporter-k78tt\" (UID: \"354094fb-89f8-4929-aae9-dc99c906d34d\") " pod="openshift-monitoring/node-exporter-k78tt"
Jul 29 03:56:50.701323 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:50.701207    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/254912df-9a41-48da-834d-7098495e379c-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-f7hkw\" (UID: \"254912df-9a41-48da-834d-7098495e379c\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-f7hkw"
Jul 29 03:56:50.701323 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:50.701237    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/254912df-9a41-48da-834d-7098495e379c-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-f7hkw\" (UID: \"254912df-9a41-48da-834d-7098495e379c\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-f7hkw"
Jul 29 03:56:50.701323 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:50.701261    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/354094fb-89f8-4929-aae9-dc99c906d34d-root\") pod \"node-exporter-k78tt\" (UID: \"354094fb-89f8-4929-aae9-dc99c906d34d\") " pod="openshift-monitoring/node-exporter-k78tt"
Jul 29 03:56:50.701323 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:50.701284    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/354094fb-89f8-4929-aae9-dc99c906d34d-node-exporter-textfile\") pod \"node-exporter-k78tt\" (UID: \"354094fb-89f8-4929-aae9-dc99c906d34d\") " pod="openshift-monitoring/node-exporter-k78tt"
Jul 29 03:56:50.701323 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:50.701317    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/254912df-9a41-48da-834d-7098495e379c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-f7hkw\" (UID: \"254912df-9a41-48da-834d-7098495e379c\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-f7hkw"
Jul 29 03:56:50.701731 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:50.701342    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/254912df-9a41-48da-834d-7098495e379c-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-f7hkw\" (UID: \"254912df-9a41-48da-834d-7098495e379c\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-f7hkw"
Jul 29 03:56:50.701731 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:50.701470    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/354094fb-89f8-4929-aae9-dc99c906d34d-sys\") pod \"node-exporter-k78tt\" (UID: \"354094fb-89f8-4929-aae9-dc99c906d34d\") " pod="openshift-monitoring/node-exporter-k78tt"
Jul 29 03:56:50.701731 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:50.701483    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/354094fb-89f8-4929-aae9-dc99c906d34d-node-exporter-wtmp\") pod \"node-exporter-k78tt\" (UID: \"354094fb-89f8-4929-aae9-dc99c906d34d\") " pod="openshift-monitoring/node-exporter-k78tt"
Jul 29 03:56:50.701731 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:50.701561    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/354094fb-89f8-4929-aae9-dc99c906d34d-node-exporter-accelerators-collector-config\") pod \"node-exporter-k78tt\" (UID: \"354094fb-89f8-4929-aae9-dc99c906d34d\") " pod="openshift-monitoring/node-exporter-k78tt"
Jul 29 03:56:50.701731 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:56:50.701568    2570 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 29 03:56:50.701731 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:56:50.701641    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/254912df-9a41-48da-834d-7098495e379c-kube-state-metrics-tls podName:254912df-9a41-48da-834d-7098495e379c nodeName:}" failed. No retries permitted until 2026-07-29 03:56:51.201623335 +0000 UTC m=+75.931754385 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/254912df-9a41-48da-834d-7098495e379c-kube-state-metrics-tls") pod "kube-state-metrics-56d9f944b4-f7hkw" (UID: "254912df-9a41-48da-834d-7098495e379c") : secret "kube-state-metrics-tls" not found
Jul 29 03:56:50.702042 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:50.701766    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/354094fb-89f8-4929-aae9-dc99c906d34d-root\") pod \"node-exporter-k78tt\" (UID: \"354094fb-89f8-4929-aae9-dc99c906d34d\") " pod="openshift-monitoring/node-exporter-k78tt"
Jul 29 03:56:50.702042 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:50.701978    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/254912df-9a41-48da-834d-7098495e379c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-f7hkw\" (UID: \"254912df-9a41-48da-834d-7098495e379c\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-f7hkw"
Jul 29 03:56:50.702042 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:50.702008    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/354094fb-89f8-4929-aae9-dc99c906d34d-metrics-client-ca\") pod \"node-exporter-k78tt\" (UID: \"354094fb-89f8-4929-aae9-dc99c906d34d\") " pod="openshift-monitoring/node-exporter-k78tt"
Jul 29 03:56:50.702188 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:50.702085    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/354094fb-89f8-4929-aae9-dc99c906d34d-node-exporter-textfile\") pod \"node-exporter-k78tt\" (UID: \"354094fb-89f8-4929-aae9-dc99c906d34d\") " pod="openshift-monitoring/node-exporter-k78tt"
Jul 29 03:56:50.702351 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:56:50.702245    2570 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 29 03:56:50.702351 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:56:50.702309    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/354094fb-89f8-4929-aae9-dc99c906d34d-node-exporter-tls podName:354094fb-89f8-4929-aae9-dc99c906d34d nodeName:}" failed. No retries permitted until 2026-07-29 03:56:51.202292992 +0000 UTC m=+75.932424069 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/354094fb-89f8-4929-aae9-dc99c906d34d-node-exporter-tls") pod "node-exporter-k78tt" (UID: "354094fb-89f8-4929-aae9-dc99c906d34d") : secret "node-exporter-tls" not found
Jul 29 03:56:50.702496 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:50.702475    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/254912df-9a41-48da-834d-7098495e379c-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-f7hkw\" (UID: \"254912df-9a41-48da-834d-7098495e379c\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-f7hkw"
Jul 29 03:56:50.704074 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:50.704050    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/254912df-9a41-48da-834d-7098495e379c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-f7hkw\" (UID: \"254912df-9a41-48da-834d-7098495e379c\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-f7hkw"
Jul 29 03:56:50.704476 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:50.704433    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/354094fb-89f8-4929-aae9-dc99c906d34d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-k78tt\" (UID: \"354094fb-89f8-4929-aae9-dc99c906d34d\") " pod="openshift-monitoring/node-exporter-k78tt"
Jul 29 03:56:50.711459 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:50.711432    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d27f2\" (UniqueName: \"kubernetes.io/projected/354094fb-89f8-4929-aae9-dc99c906d34d-kube-api-access-d27f2\") pod \"node-exporter-k78tt\" (UID: \"354094fb-89f8-4929-aae9-dc99c906d34d\") " pod="openshift-monitoring/node-exporter-k78tt"
Jul 29 03:56:50.711950 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:50.711926    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm5gj\" (UniqueName: \"kubernetes.io/projected/254912df-9a41-48da-834d-7098495e379c-kube-api-access-dm5gj\") pod \"kube-state-metrics-56d9f944b4-f7hkw\" (UID: \"254912df-9a41-48da-834d-7098495e379c\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-f7hkw"
Jul 29 03:56:51.104006 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:51.103973    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/364ce126-2396-443f-a420-7be5f74d1494-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-bwtm8\" (UID: \"364ce126-2396-443f-a420-7be5f74d1494\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-bwtm8"
Jul 29 03:56:51.106428 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:51.106399    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/364ce126-2396-443f-a420-7be5f74d1494-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-bwtm8\" (UID: \"364ce126-2396-443f-a420-7be5f74d1494\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-bwtm8"
Jul 29 03:56:51.205013 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:51.204989    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/354094fb-89f8-4929-aae9-dc99c906d34d-node-exporter-tls\") pod \"node-exporter-k78tt\" (UID: \"354094fb-89f8-4929-aae9-dc99c906d34d\") " pod="openshift-monitoring/node-exporter-k78tt"
Jul 29 03:56:51.205078 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:51.205035    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/254912df-9a41-48da-834d-7098495e379c-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-f7hkw\" (UID: \"254912df-9a41-48da-834d-7098495e379c\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-f7hkw"
Jul 29 03:56:51.207374 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:51.207349    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/354094fb-89f8-4929-aae9-dc99c906d34d-node-exporter-tls\") pod \"node-exporter-k78tt\" (UID: \"354094fb-89f8-4929-aae9-dc99c906d34d\") " pod="openshift-monitoring/node-exporter-k78tt"
Jul 29 03:56:51.207464 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:51.207446    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/254912df-9a41-48da-834d-7098495e379c-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-f7hkw\" (UID: \"254912df-9a41-48da-834d-7098495e379c\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-f7hkw"
Jul 29 03:56:51.389965 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:51.389899    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-bwtm8"
Jul 29 03:56:51.412991 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:51.412969    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-f7hkw"
Jul 29 03:56:51.425623 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:51.425601    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-k78tt"
Jul 29 03:56:51.438220 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:56:51.438183    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod354094fb_89f8_4929_aae9_dc99c906d34d.slice/crio-894924d6e62479215fc96a336bd9a223c1855ebf93446a9ba240cfc9016b2bf1 WatchSource:0}: Error finding container 894924d6e62479215fc96a336bd9a223c1855ebf93446a9ba240cfc9016b2bf1: Status 404 returned error can't find the container with id 894924d6e62479215fc96a336bd9a223c1855ebf93446a9ba240cfc9016b2bf1
Jul 29 03:56:51.544353 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:51.543916    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-bwtm8"]
Jul 29 03:56:51.548384 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:56:51.548353    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod364ce126_2396_443f_a420_7be5f74d1494.slice/crio-6c5c48038355c98f8566864bf878b351183076c19e567476dd36eef94cf70d14 WatchSource:0}: Error finding container 6c5c48038355c98f8566864bf878b351183076c19e567476dd36eef94cf70d14: Status 404 returned error can't find the container with id 6c5c48038355c98f8566864bf878b351183076c19e567476dd36eef94cf70d14
Jul 29 03:56:51.568303 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:51.568280    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 29 03:56:51.573238 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:51.573216    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-f7hkw"]
Jul 29 03:56:51.573332 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:51.573313    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 29 03:56:51.576753 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:51.576737    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 29 03:56:51.576872 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:51.576752    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 29 03:56:51.576932 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:51.576900    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 29 03:56:51.577003 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:51.576989    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 29 03:56:51.577178 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:51.577161    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 29 03:56:51.577371 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:51.577358    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-gklfl\""
Jul 29 03:56:51.578484 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:51.578251    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 29 03:56:51.578484 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:51.578423    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 29 03:56:51.578772 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:51.578568    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 29 03:56:51.578772 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:51.578688    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 29 03:56:51.579245 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:56:51.579223    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod254912df_9a41_48da_834d_7098495e379c.slice/crio-69c249ee8a26fed27912f5408ba0e85cb340ca1fa9eaf127981aa655d8146d81 WatchSource:0}: Error finding container 69c249ee8a26fed27912f5408ba0e85cb340ca1fa9eaf127981aa655d8146d81: Status 404 returned error can't find the container with id 69c249ee8a26fed27912f5408ba0e85cb340ca1fa9eaf127981aa655d8146d81
Jul 29 03:56:51.590910 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:51.590891    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 29 03:56:51.609577 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:51.609297    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1416a1fd-24a7-4d3c-93e7-5d8198df9cac-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1416a1fd-24a7-4d3c-93e7-5d8198df9cac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 03:56:51.609577 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:51.609355    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1416a1fd-24a7-4d3c-93e7-5d8198df9cac-config-out\") pod \"alertmanager-main-0\" (UID: \"1416a1fd-24a7-4d3c-93e7-5d8198df9cac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 03:56:51.609577 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:51.609392    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp2xb\" (UniqueName: \"kubernetes.io/projected/1416a1fd-24a7-4d3c-93e7-5d8198df9cac-kube-api-access-wp2xb\") pod \"alertmanager-main-0\" (UID: \"1416a1fd-24a7-4d3c-93e7-5d8198df9cac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 03:56:51.609577 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:51.609561    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1416a1fd-24a7-4d3c-93e7-5d8198df9cac-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1416a1fd-24a7-4d3c-93e7-5d8198df9cac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 03:56:51.609843 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:51.609608    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1416a1fd-24a7-4d3c-93e7-5d8198df9cac-web-config\") pod \"alertmanager-main-0\" (UID: \"1416a1fd-24a7-4d3c-93e7-5d8198df9cac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 03:56:51.609843 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:51.609645    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1416a1fd-24a7-4d3c-93e7-5d8198df9cac-config-volume\") pod \"alertmanager-main-0\" (UID: \"1416a1fd-24a7-4d3c-93e7-5d8198df9cac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 03:56:51.609843 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:51.609677    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1416a1fd-24a7-4d3c-93e7-5d8198df9cac-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1416a1fd-24a7-4d3c-93e7-5d8198df9cac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 03:56:51.609843 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:51.609738    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1416a1fd-24a7-4d3c-93e7-5d8198df9cac-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1416a1fd-24a7-4d3c-93e7-5d8198df9cac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 03:56:51.609843 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:51.609767    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1416a1fd-24a7-4d3c-93e7-5d8198df9cac-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1416a1fd-24a7-4d3c-93e7-5d8198df9cac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 03:56:51.609843 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:51.609805    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1416a1fd-24a7-4d3c-93e7-5d8198df9cac-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1416a1fd-24a7-4d3c-93e7-5d8198df9cac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 03:56:51.610099 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:51.609865    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1416a1fd-24a7-4d3c-93e7-5d8198df9cac-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1416a1fd-24a7-4d3c-93e7-5d8198df9cac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 03:56:51.610099 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:51.609908    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1416a1fd-24a7-4d3c-93e7-5d8198df9cac-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1416a1fd-24a7-4d3c-93e7-5d8198df9cac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 03:56:51.610099 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:51.610034    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1416a1fd-24a7-4d3c-93e7-5d8198df9cac-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1416a1fd-24a7-4d3c-93e7-5d8198df9cac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 03:56:51.711319 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:51.711282    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1416a1fd-24a7-4d3c-93e7-5d8198df9cac-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1416a1fd-24a7-4d3c-93e7-5d8198df9cac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 03:56:51.711319 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:51.711317    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1416a1fd-24a7-4d3c-93e7-5d8198df9cac-web-config\") pod \"alertmanager-main-0\" (UID: \"1416a1fd-24a7-4d3c-93e7-5d8198df9cac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 03:56:51.711479 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:51.711334    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1416a1fd-24a7-4d3c-93e7-5d8198df9cac-config-volume\") pod \"alertmanager-main-0\" (UID: \"1416a1fd-24a7-4d3c-93e7-5d8198df9cac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 03:56:51.711479 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:51.711387    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1416a1fd-24a7-4d3c-93e7-5d8198df9cac-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1416a1fd-24a7-4d3c-93e7-5d8198df9cac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 03:56:51.711479 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:51.711446    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1416a1fd-24a7-4d3c-93e7-5d8198df9cac-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1416a1fd-24a7-4d3c-93e7-5d8198df9cac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 03:56:51.711667 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:51.711639    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1416a1fd-24a7-4d3c-93e7-5d8198df9cac-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1416a1fd-24a7-4d3c-93e7-5d8198df9cac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 03:56:51.711734 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:51.711696    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1416a1fd-24a7-4d3c-93e7-5d8198df9cac-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1416a1fd-24a7-4d3c-93e7-5d8198df9cac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 03:56:51.711788 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:51.711733    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1416a1fd-24a7-4d3c-93e7-5d8198df9cac-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1416a1fd-24a7-4d3c-93e7-5d8198df9cac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 03:56:51.711788 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:51.711765    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1416a1fd-24a7-4d3c-93e7-5d8198df9cac-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1416a1fd-24a7-4d3c-93e7-5d8198df9cac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 03:56:51.711917 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:51.711806    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1416a1fd-24a7-4d3c-93e7-5d8198df9cac-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1416a1fd-24a7-4d3c-93e7-5d8198df9cac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 03:56:51.711917 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:51.711850    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1416a1fd-24a7-4d3c-93e7-5d8198df9cac-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1416a1fd-24a7-4d3c-93e7-5d8198df9cac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 03:56:51.711917 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:51.711881    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1416a1fd-24a7-4d3c-93e7-5d8198df9cac-config-out\") pod \"alertmanager-main-0\" (UID: \"1416a1fd-24a7-4d3c-93e7-5d8198df9cac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 03:56:51.711917 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:51.711907    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp2xb\" (UniqueName: \"kubernetes.io/projected/1416a1fd-24a7-4d3c-93e7-5d8198df9cac-kube-api-access-wp2xb\") pod \"alertmanager-main-0\" (UID: \"1416a1fd-24a7-4d3c-93e7-5d8198df9cac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 03:56:51.712616 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:51.712320    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1416a1fd-24a7-4d3c-93e7-5d8198df9cac-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1416a1fd-24a7-4d3c-93e7-5d8198df9cac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 03:56:51.712616 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:56:51.712353    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1416a1fd-24a7-4d3c-93e7-5d8198df9cac-alertmanager-trusted-ca-bundle podName:1416a1fd-24a7-4d3c-93e7-5d8198df9cac nodeName:}" failed. No retries permitted until 2026-07-29 03:56:52.212333631 +0000 UTC m=+76.942464683 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/1416a1fd-24a7-4d3c-93e7-5d8198df9cac-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "1416a1fd-24a7-4d3c-93e7-5d8198df9cac") : configmap references non-existent config key: ca-bundle.crt
Jul 29 03:56:51.712791 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:51.712643    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1416a1fd-24a7-4d3c-93e7-5d8198df9cac-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1416a1fd-24a7-4d3c-93e7-5d8198df9cac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 03:56:51.712791 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:56:51.712740    2570 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Jul 29 03:56:51.712791 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:56:51.712790    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1416a1fd-24a7-4d3c-93e7-5d8198df9cac-secret-alertmanager-main-tls podName:1416a1fd-24a7-4d3c-93e7-5d8198df9cac nodeName:}" failed. No retries permitted until 2026-07-29 03:56:52.212772376 +0000 UTC m=+76.942903438 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/1416a1fd-24a7-4d3c-93e7-5d8198df9cac-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "1416a1fd-24a7-4d3c-93e7-5d8198df9cac") : secret "alertmanager-main-tls" not found
Jul 29 03:56:51.714609 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:51.714582    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1416a1fd-24a7-4d3c-93e7-5d8198df9cac-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1416a1fd-24a7-4d3c-93e7-5d8198df9cac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 03:56:51.714609 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:51.714755    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1416a1fd-24a7-4d3c-93e7-5d8198df9cac-config-volume\") pod \"alertmanager-main-0\" (UID: \"1416a1fd-24a7-4d3c-93e7-5d8198df9cac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 03:56:51.714609 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:51.714885    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1416a1fd-24a7-4d3c-93e7-5d8198df9cac-web-config\") pod \"alertmanager-main-0\" (UID: \"1416a1fd-24a7-4d3c-93e7-5d8198df9cac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 03:56:51.714609 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:51.714967    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1416a1fd-24a7-4d3c-93e7-5d8198df9cac-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1416a1fd-24a7-4d3c-93e7-5d8198df9cac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 03:56:51.714609 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:51.715068    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1416a1fd-24a7-4d3c-93e7-5d8198df9cac-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1416a1fd-24a7-4d3c-93e7-5d8198df9cac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 03:56:51.714609 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:51.715104    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1416a1fd-24a7-4d3c-93e7-5d8198df9cac-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1416a1fd-24a7-4d3c-93e7-5d8198df9cac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 03:56:51.715721 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:51.715659    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1416a1fd-24a7-4d3c-93e7-5d8198df9cac-config-out\") pod \"alertmanager-main-0\" (UID: \"1416a1fd-24a7-4d3c-93e7-5d8198df9cac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 03:56:51.716398 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:51.716374    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1416a1fd-24a7-4d3c-93e7-5d8198df9cac-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1416a1fd-24a7-4d3c-93e7-5d8198df9cac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 03:56:51.720666 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:51.720646    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp2xb\" (UniqueName: \"kubernetes.io/projected/1416a1fd-24a7-4d3c-93e7-5d8198df9cac-kube-api-access-wp2xb\") pod \"alertmanager-main-0\" (UID: \"1416a1fd-24a7-4d3c-93e7-5d8198df9cac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 03:56:52.217666 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:52.217632    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1416a1fd-24a7-4d3c-93e7-5d8198df9cac-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1416a1fd-24a7-4d3c-93e7-5d8198df9cac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 03:56:52.217984 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:52.217727    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1416a1fd-24a7-4d3c-93e7-5d8198df9cac-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1416a1fd-24a7-4d3c-93e7-5d8198df9cac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 03:56:52.218555 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:52.218524    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1416a1fd-24a7-4d3c-93e7-5d8198df9cac-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1416a1fd-24a7-4d3c-93e7-5d8198df9cac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 03:56:52.220874 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:52.220848    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1416a1fd-24a7-4d3c-93e7-5d8198df9cac-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1416a1fd-24a7-4d3c-93e7-5d8198df9cac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 03:56:52.368151 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:52.368107    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k78tt" event={"ID":"354094fb-89f8-4929-aae9-dc99c906d34d","Type":"ContainerStarted","Data":"894924d6e62479215fc96a336bd9a223c1855ebf93446a9ba240cfc9016b2bf1"}
Jul 29 03:56:52.369610 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:52.369574    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-f7hkw" event={"ID":"254912df-9a41-48da-834d-7098495e379c","Type":"ContainerStarted","Data":"69c249ee8a26fed27912f5408ba0e85cb340ca1fa9eaf127981aa655d8146d81"}
Jul 29 03:56:52.372525 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:52.372498    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-bwtm8" event={"ID":"364ce126-2396-443f-a420-7be5f74d1494","Type":"ContainerStarted","Data":"4a23d7f36ecef69c96120caa03b25ca1311997953a28d3a6b9ebc7d73cee45ba"}
Jul 29 03:56:52.372657 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:52.372549    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-bwtm8" event={"ID":"364ce126-2396-443f-a420-7be5f74d1494","Type":"ContainerStarted","Data":"76ded021b78279c392d9303fb6ea34c4a33cc432fad5498e24e2cd369f0e2c1e"}
Jul 29 03:56:52.372657 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:52.372569    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-bwtm8" event={"ID":"364ce126-2396-443f-a420-7be5f74d1494","Type":"ContainerStarted","Data":"6c5c48038355c98f8566864bf878b351183076c19e567476dd36eef94cf70d14"}
Jul 29 03:56:52.499630 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:52.499559    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 29 03:56:53.205788 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:53.205755    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 29 03:56:53.210529 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:56:53.210479    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1416a1fd_24a7_4d3c_93e7_5d8198df9cac.slice/crio-b11082dfe34250e06f458e8014d56a6cd728aef1b9e89f5d37e2f7c52f4e217a WatchSource:0}: Error finding container b11082dfe34250e06f458e8014d56a6cd728aef1b9e89f5d37e2f7c52f4e217a: Status 404 returned error can't find the container with id b11082dfe34250e06f458e8014d56a6cd728aef1b9e89f5d37e2f7c52f4e217a
Jul 29 03:56:53.377694 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:53.377657    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-bwtm8" event={"ID":"364ce126-2396-443f-a420-7be5f74d1494","Type":"ContainerStarted","Data":"e544ec9fb51c73ffc8f8e2878bb5f02e046bc65b4496d624057a610899185d7c"}
Jul 29 03:56:53.379419 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:53.379393    2570 generic.go:356] "Generic (PLEG): container finished" podID="354094fb-89f8-4929-aae9-dc99c906d34d" containerID="e664c8dd1951fb1cef2546e1184be410708e43ef51ffbf6a51eaef82483547dd" exitCode=0
Jul 29 03:56:53.379546 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:53.379469    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k78tt" event={"ID":"354094fb-89f8-4929-aae9-dc99c906d34d","Type":"ContainerDied","Data":"e664c8dd1951fb1cef2546e1184be410708e43ef51ffbf6a51eaef82483547dd"}
Jul 29 03:56:53.381721 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:53.381695    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-f7hkw" event={"ID":"254912df-9a41-48da-834d-7098495e379c","Type":"ContainerStarted","Data":"9618d278802d8cd34f4ac9dcea470f1cc9984b2be9a057097c934896263c8c42"}
Jul 29 03:56:53.381847 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:53.381750    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-f7hkw" event={"ID":"254912df-9a41-48da-834d-7098495e379c","Type":"ContainerStarted","Data":"382a3cf24e4c2ccd3eb90e9981e31fb2156b2cb0e3719c56cdd6e167dd62ea61"}
Jul 29 03:56:53.381847 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:53.381774    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-f7hkw" event={"ID":"254912df-9a41-48da-834d-7098495e379c","Type":"ContainerStarted","Data":"34c7fcb3daf25fd250b56c9aaedc5a089bd4ef5c6ce2ee00130b740274112c62"}
Jul 29 03:56:53.383063 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:53.382990    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1416a1fd-24a7-4d3c-93e7-5d8198df9cac","Type":"ContainerStarted","Data":"b11082dfe34250e06f458e8014d56a6cd728aef1b9e89f5d37e2f7c52f4e217a"}
Jul 29 03:56:53.398055 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:53.397947    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-bwtm8" podStartSLOduration=2.014165243 podStartE2EDuration="3.397929343s" totalImagesPullingTime="1.3837641s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 03:56:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:56:53.397241899 +0000 UTC m=+78.127372970" watchObservedRunningTime="2026-07-29 03:56:53.397929343 +0000 UTC m=+78.128060414"
Jul 29 03:56:53.427010 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:53.426965    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-f7hkw" podStartSLOduration=1.950927041 podStartE2EDuration="3.426949772s" totalImagesPullingTime="1.476022731s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 03:56:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:56:53.426291691 +0000 UTC m=+78.156422763" watchObservedRunningTime="2026-07-29 03:56:53.426949772 +0000 UTC m=+78.157080842"
Jul 29 03:56:54.365531 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:54.365508    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-749c48b96f-q7lv6"
Jul 29 03:56:54.369334 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:54.369310    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-749c48b96f-q7lv6"
Jul 29 03:56:54.388699 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:54.388673    2570 generic.go:356] "Generic (PLEG): container finished" podID="1416a1fd-24a7-4d3c-93e7-5d8198df9cac" containerID="2cd7f880df8ef32063cb0b65fc49c25e974f7e7c517ca68efb6744dfca0084fd" exitCode=0
Jul 29 03:56:54.388803 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:54.388714    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1416a1fd-24a7-4d3c-93e7-5d8198df9cac","Type":"ContainerDied","Data":"2cd7f880df8ef32063cb0b65fc49c25e974f7e7c517ca68efb6744dfca0084fd"}
Jul 29 03:56:54.391069 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:54.391039    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k78tt" event={"ID":"354094fb-89f8-4929-aae9-dc99c906d34d","Type":"ContainerStarted","Data":"e34f1dbc1babc3c80e59dba438d80f137abc61a78edd50998022e5184efff45b"}
Jul 29 03:56:54.391165 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:54.391086    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k78tt" event={"ID":"354094fb-89f8-4929-aae9-dc99c906d34d","Type":"ContainerStarted","Data":"cd5c785f0cdbf6e0064464a36a95d2a77da6079bf4f46590b7a55b227a704aef"}
Jul 29 03:56:54.420431 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:54.419784    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-k78tt" podStartSLOduration=3.310816315 podStartE2EDuration="4.419766529s" totalImagesPullingTime="1.006256003s" totalInitContainerRuntime="102.694211ms" isStatefulPod=true podCreationTimestamp="2026-07-29 03:56:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:56:54.419518875 +0000 UTC m=+79.149649945" watchObservedRunningTime="2026-07-29 03:56:54.419766529 +0000 UTC m=+79.149897601"
Jul 29 03:56:54.479778 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:54.479747    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-8686848888-grwsz"]
Jul 29 03:56:54.483248 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:54.483231    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-8686848888-grwsz"
Jul 29 03:56:54.487589 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:54.487530    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 29 03:56:54.487721 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:54.487692    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 29 03:56:54.487721 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:54.487715    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 29 03:56:54.487868 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:54.487753    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-9psrk\""
Jul 29 03:56:54.487868 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:54.487762    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 29 03:56:54.488056 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:54.488036    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 29 03:56:54.488123 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:54.488097    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-7jrg03ib6ch0f\""
Jul 29 03:56:54.497731 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:54.497711    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-8686848888-grwsz"]
Jul 29 03:56:54.539649 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:54.539626    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4b2f4957-a89a-4c88-beff-248aca3c6c89-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-8686848888-grwsz\" (UID: \"4b2f4957-a89a-4c88-beff-248aca3c6c89\") " pod="openshift-monitoring/thanos-querier-8686848888-grwsz"
Jul 29 03:56:54.539785 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:54.539668    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4b2f4957-a89a-4c88-beff-248aca3c6c89-secret-thanos-querier-tls\") pod \"thanos-querier-8686848888-grwsz\" (UID: \"4b2f4957-a89a-4c88-beff-248aca3c6c89\") " pod="openshift-monitoring/thanos-querier-8686848888-grwsz"
Jul 29 03:56:54.539785 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:54.539693    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4b2f4957-a89a-4c88-beff-248aca3c6c89-metrics-client-ca\") pod \"thanos-querier-8686848888-grwsz\" (UID: \"4b2f4957-a89a-4c88-beff-248aca3c6c89\") " pod="openshift-monitoring/thanos-querier-8686848888-grwsz"
Jul 29 03:56:54.539785 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:54.539760    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4b2f4957-a89a-4c88-beff-248aca3c6c89-secret-grpc-tls\") pod \"thanos-querier-8686848888-grwsz\" (UID: \"4b2f4957-a89a-4c88-beff-248aca3c6c89\") " pod="openshift-monitoring/thanos-querier-8686848888-grwsz"
Jul 29 03:56:54.539974 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:54.539805    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4b2f4957-a89a-4c88-beff-248aca3c6c89-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-8686848888-grwsz\" (UID: \"4b2f4957-a89a-4c88-beff-248aca3c6c89\") " pod="openshift-monitoring/thanos-querier-8686848888-grwsz"
Jul 29 03:56:54.539974 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:54.539877    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4b2f4957-a89a-4c88-beff-248aca3c6c89-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-8686848888-grwsz\" (UID: \"4b2f4957-a89a-4c88-beff-248aca3c6c89\") " pod="openshift-monitoring/thanos-querier-8686848888-grwsz"
Jul 29 03:56:54.539974 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:54.539913    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4b2f4957-a89a-4c88-beff-248aca3c6c89-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-8686848888-grwsz\" (UID: \"4b2f4957-a89a-4c88-beff-248aca3c6c89\") " pod="openshift-monitoring/thanos-querier-8686848888-grwsz"
Jul 29 03:56:54.539974 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:54.539945    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlwdh\" (UniqueName: \"kubernetes.io/projected/4b2f4957-a89a-4c88-beff-248aca3c6c89-kube-api-access-tlwdh\") pod \"thanos-querier-8686848888-grwsz\" (UID: \"4b2f4957-a89a-4c88-beff-248aca3c6c89\") " pod="openshift-monitoring/thanos-querier-8686848888-grwsz"
Jul 29 03:56:54.641160 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:54.641135    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4b2f4957-a89a-4c88-beff-248aca3c6c89-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-8686848888-grwsz\" (UID: \"4b2f4957-a89a-4c88-beff-248aca3c6c89\") " pod="openshift-monitoring/thanos-querier-8686848888-grwsz"
Jul 29 03:56:54.641280 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:54.641169    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlwdh\" (UniqueName: \"kubernetes.io/projected/4b2f4957-a89a-4c88-beff-248aca3c6c89-kube-api-access-tlwdh\") pod \"thanos-querier-8686848888-grwsz\" (UID: \"4b2f4957-a89a-4c88-beff-248aca3c6c89\") " pod="openshift-monitoring/thanos-querier-8686848888-grwsz"
Jul 29 03:56:54.641280 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:54.641199    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4b2f4957-a89a-4c88-beff-248aca3c6c89-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-8686848888-grwsz\" (UID: \"4b2f4957-a89a-4c88-beff-248aca3c6c89\") " pod="openshift-monitoring/thanos-querier-8686848888-grwsz"
Jul 29 03:56:54.641280 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:54.641221    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4b2f4957-a89a-4c88-beff-248aca3c6c89-secret-thanos-querier-tls\") pod \"thanos-querier-8686848888-grwsz\" (UID: \"4b2f4957-a89a-4c88-beff-248aca3c6c89\") " pod="openshift-monitoring/thanos-querier-8686848888-grwsz"
Jul 29 03:56:54.641280 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:54.641245    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4b2f4957-a89a-4c88-beff-248aca3c6c89-metrics-client-ca\") pod \"thanos-querier-8686848888-grwsz\" (UID: \"4b2f4957-a89a-4c88-beff-248aca3c6c89\") " pod="openshift-monitoring/thanos-querier-8686848888-grwsz"
Jul 29 03:56:54.641280 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:54.641264    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4b2f4957-a89a-4c88-beff-248aca3c6c89-secret-grpc-tls\") pod \"thanos-querier-8686848888-grwsz\" (UID: \"4b2f4957-a89a-4c88-beff-248aca3c6c89\") " pod="openshift-monitoring/thanos-querier-8686848888-grwsz"
Jul 29 03:56:54.641527 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:54.641286    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4b2f4957-a89a-4c88-beff-248aca3c6c89-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-8686848888-grwsz\" (UID: \"4b2f4957-a89a-4c88-beff-248aca3c6c89\") " pod="openshift-monitoring/thanos-querier-8686848888-grwsz"
Jul 29 03:56:54.641527 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:54.641313    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4b2f4957-a89a-4c88-beff-248aca3c6c89-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-8686848888-grwsz\" (UID: \"4b2f4957-a89a-4c88-beff-248aca3c6c89\") " pod="openshift-monitoring/thanos-querier-8686848888-grwsz"
Jul 29 03:56:54.642336 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:54.642308    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4b2f4957-a89a-4c88-beff-248aca3c6c89-metrics-client-ca\") pod \"thanos-querier-8686848888-grwsz\" (UID: \"4b2f4957-a89a-4c88-beff-248aca3c6c89\") " pod="openshift-monitoring/thanos-querier-8686848888-grwsz"
Jul 29 03:56:54.643911 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:54.643880    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4b2f4957-a89a-4c88-beff-248aca3c6c89-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-8686848888-grwsz\" (UID: \"4b2f4957-a89a-4c88-beff-248aca3c6c89\") " pod="openshift-monitoring/thanos-querier-8686848888-grwsz"
Jul 29 03:56:54.644025 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:54.644006    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4b2f4957-a89a-4c88-beff-248aca3c6c89-secret-grpc-tls\") pod \"thanos-querier-8686848888-grwsz\" (UID: \"4b2f4957-a89a-4c88-beff-248aca3c6c89\") " pod="openshift-monitoring/thanos-querier-8686848888-grwsz"
Jul 29 03:56:54.644205 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:54.644184    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4b2f4957-a89a-4c88-beff-248aca3c6c89-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-8686848888-grwsz\" (UID: \"4b2f4957-a89a-4c88-beff-248aca3c6c89\") " pod="openshift-monitoring/thanos-querier-8686848888-grwsz"
Jul 29 03:56:54.644286 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:54.644264    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4b2f4957-a89a-4c88-beff-248aca3c6c89-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-8686848888-grwsz\" (UID: \"4b2f4957-a89a-4c88-beff-248aca3c6c89\") " pod="openshift-monitoring/thanos-querier-8686848888-grwsz"
Jul 29 03:56:54.644354 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:54.644339    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4b2f4957-a89a-4c88-beff-248aca3c6c89-secret-thanos-querier-tls\") pod \"thanos-querier-8686848888-grwsz\" (UID: \"4b2f4957-a89a-4c88-beff-248aca3c6c89\") " pod="openshift-monitoring/thanos-querier-8686848888-grwsz"
Jul 29 03:56:54.644410 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:54.644366    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4b2f4957-a89a-4c88-beff-248aca3c6c89-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-8686848888-grwsz\" (UID: \"4b2f4957-a89a-4c88-beff-248aca3c6c89\") " pod="openshift-monitoring/thanos-querier-8686848888-grwsz"
Jul 29 03:56:54.648764 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:54.648743    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlwdh\" (UniqueName: \"kubernetes.io/projected/4b2f4957-a89a-4c88-beff-248aca3c6c89-kube-api-access-tlwdh\") pod \"thanos-querier-8686848888-grwsz\" (UID: \"4b2f4957-a89a-4c88-beff-248aca3c6c89\") " pod="openshift-monitoring/thanos-querier-8686848888-grwsz"
Jul 29 03:56:54.774878 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:54.774848    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7f5b4b5cf8-nvzjr"]
Jul 29 03:56:54.779101 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:54.779085    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7f5b4b5cf8-nvzjr"
Jul 29 03:56:54.781939 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:54.781906    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-748doj46mgu7s\""
Jul 29 03:56:54.781939 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:54.781909    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 29 03:56:54.782116 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:54.781988    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-5zpl5\""
Jul 29 03:56:54.782116 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:54.781999    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 29 03:56:54.782116 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:54.782027    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 29 03:56:54.782116 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:54.782077    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 29 03:56:54.787222 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:54.787200    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7f5b4b5cf8-nvzjr"]
Jul 29 03:56:54.793117 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:54.793097    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-8686848888-grwsz"
Jul 29 03:56:54.843053 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:54.843022    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d2bx\" (UniqueName: \"kubernetes.io/projected/aca3b4b2-ca5d-4712-9b80-d9d2a41380e6-kube-api-access-6d2bx\") pod \"metrics-server-7f5b4b5cf8-nvzjr\" (UID: \"aca3b4b2-ca5d-4712-9b80-d9d2a41380e6\") " pod="openshift-monitoring/metrics-server-7f5b4b5cf8-nvzjr"
Jul 29 03:56:54.843187 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:54.843095    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/aca3b4b2-ca5d-4712-9b80-d9d2a41380e6-audit-log\") pod \"metrics-server-7f5b4b5cf8-nvzjr\" (UID: \"aca3b4b2-ca5d-4712-9b80-d9d2a41380e6\") " pod="openshift-monitoring/metrics-server-7f5b4b5cf8-nvzjr"
Jul 29 03:56:54.843268 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:54.843173    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aca3b4b2-ca5d-4712-9b80-d9d2a41380e6-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7f5b4b5cf8-nvzjr\" (UID: \"aca3b4b2-ca5d-4712-9b80-d9d2a41380e6\") " pod="openshift-monitoring/metrics-server-7f5b4b5cf8-nvzjr"
Jul 29 03:56:54.843316 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:54.843266    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/aca3b4b2-ca5d-4712-9b80-d9d2a41380e6-metrics-server-audit-profiles\") pod \"metrics-server-7f5b4b5cf8-nvzjr\" (UID: \"aca3b4b2-ca5d-4712-9b80-d9d2a41380e6\") " pod="openshift-monitoring/metrics-server-7f5b4b5cf8-nvzjr"
Jul 29 03:56:54.843367 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:54.843319    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/aca3b4b2-ca5d-4712-9b80-d9d2a41380e6-secret-metrics-server-tls\") pod \"metrics-server-7f5b4b5cf8-nvzjr\" (UID: \"aca3b4b2-ca5d-4712-9b80-d9d2a41380e6\") " pod="openshift-monitoring/metrics-server-7f5b4b5cf8-nvzjr"
Jul 29 03:56:54.843367 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:54.843355    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/aca3b4b2-ca5d-4712-9b80-d9d2a41380e6-client-ca-bundle\") pod \"metrics-server-7f5b4b5cf8-nvzjr\" (UID: \"aca3b4b2-ca5d-4712-9b80-d9d2a41380e6\") " pod="openshift-monitoring/metrics-server-7f5b4b5cf8-nvzjr"
Jul 29 03:56:54.843464 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:54.843393    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/aca3b4b2-ca5d-4712-9b80-d9d2a41380e6-secret-metrics-server-client-certs\") pod \"metrics-server-7f5b4b5cf8-nvzjr\" (UID: \"aca3b4b2-ca5d-4712-9b80-d9d2a41380e6\") " pod="openshift-monitoring/metrics-server-7f5b4b5cf8-nvzjr"
Jul 29 03:56:54.923299 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:54.923270    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-8686848888-grwsz"]
Jul 29 03:56:54.927630 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:56:54.927579    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b2f4957_a89a_4c88_beff_248aca3c6c89.slice/crio-94e25242923ce20b3a600652502027a39c83b91b37698f34a90f5e3a05e137b5 WatchSource:0}: Error finding container 94e25242923ce20b3a600652502027a39c83b91b37698f34a90f5e3a05e137b5: Status 404 returned error can't find the container with id 94e25242923ce20b3a600652502027a39c83b91b37698f34a90f5e3a05e137b5
Jul 29 03:56:54.944511 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:54.944487    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/aca3b4b2-ca5d-4712-9b80-d9d2a41380e6-audit-log\") pod \"metrics-server-7f5b4b5cf8-nvzjr\" (UID: \"aca3b4b2-ca5d-4712-9b80-d9d2a41380e6\") " pod="openshift-monitoring/metrics-server-7f5b4b5cf8-nvzjr"
Jul 29 03:56:54.944632 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:54.944538    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aca3b4b2-ca5d-4712-9b80-d9d2a41380e6-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7f5b4b5cf8-nvzjr\" (UID: \"aca3b4b2-ca5d-4712-9b80-d9d2a41380e6\") " pod="openshift-monitoring/metrics-server-7f5b4b5cf8-nvzjr"
Jul 29 03:56:54.944632 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:54.944588    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/aca3b4b2-ca5d-4712-9b80-d9d2a41380e6-metrics-server-audit-profiles\") pod \"metrics-server-7f5b4b5cf8-nvzjr\" (UID: \"aca3b4b2-ca5d-4712-9b80-d9d2a41380e6\") " pod="openshift-monitoring/metrics-server-7f5b4b5cf8-nvzjr"
Jul 29 03:56:54.944747 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:54.944632    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/aca3b4b2-ca5d-4712-9b80-d9d2a41380e6-secret-metrics-server-tls\") pod \"metrics-server-7f5b4b5cf8-nvzjr\" (UID: \"aca3b4b2-ca5d-4712-9b80-d9d2a41380e6\") " pod="openshift-monitoring/metrics-server-7f5b4b5cf8-nvzjr"
Jul 29 03:56:54.944747 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:54.944692    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/aca3b4b2-ca5d-4712-9b80-d9d2a41380e6-client-ca-bundle\") pod \"metrics-server-7f5b4b5cf8-nvzjr\" (UID: \"aca3b4b2-ca5d-4712-9b80-d9d2a41380e6\") " pod="openshift-monitoring/metrics-server-7f5b4b5cf8-nvzjr"
Jul 29 03:56:54.944747 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:54.944735    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/aca3b4b2-ca5d-4712-9b80-d9d2a41380e6-secret-metrics-server-client-certs\") pod \"metrics-server-7f5b4b5cf8-nvzjr\" (UID: \"aca3b4b2-ca5d-4712-9b80-d9d2a41380e6\") " pod="openshift-monitoring/metrics-server-7f5b4b5cf8-nvzjr"
Jul 29 03:56:54.944923 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:54.944770    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d2bx\" (UniqueName: \"kubernetes.io/projected/aca3b4b2-ca5d-4712-9b80-d9d2a41380e6-kube-api-access-6d2bx\") pod \"metrics-server-7f5b4b5cf8-nvzjr\" (UID: \"aca3b4b2-ca5d-4712-9b80-d9d2a41380e6\") " pod="openshift-monitoring/metrics-server-7f5b4b5cf8-nvzjr"
Jul 29 03:56:54.944974 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:54.944939    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/aca3b4b2-ca5d-4712-9b80-d9d2a41380e6-audit-log\") pod \"metrics-server-7f5b4b5cf8-nvzjr\" (UID: \"aca3b4b2-ca5d-4712-9b80-d9d2a41380e6\") " pod="openshift-monitoring/metrics-server-7f5b4b5cf8-nvzjr"
Jul 29 03:56:54.945332 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:54.945284    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aca3b4b2-ca5d-4712-9b80-d9d2a41380e6-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7f5b4b5cf8-nvzjr\" (UID: \"aca3b4b2-ca5d-4712-9b80-d9d2a41380e6\") " pod="openshift-monitoring/metrics-server-7f5b4b5cf8-nvzjr"
Jul 29 03:56:54.946626 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:54.946600    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/aca3b4b2-ca5d-4712-9b80-d9d2a41380e6-metrics-server-audit-profiles\") pod \"metrics-server-7f5b4b5cf8-nvzjr\" (UID: \"aca3b4b2-ca5d-4712-9b80-d9d2a41380e6\") " pod="openshift-monitoring/metrics-server-7f5b4b5cf8-nvzjr"
Jul 29 03:56:54.947657 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:54.947517    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/aca3b4b2-ca5d-4712-9b80-d9d2a41380e6-secret-metrics-server-tls\") pod \"metrics-server-7f5b4b5cf8-nvzjr\" (UID: \"aca3b4b2-ca5d-4712-9b80-d9d2a41380e6\") " pod="openshift-monitoring/metrics-server-7f5b4b5cf8-nvzjr"
Jul 29 03:56:54.947657 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:54.947643    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/aca3b4b2-ca5d-4712-9b80-d9d2a41380e6-secret-metrics-server-client-certs\") pod \"metrics-server-7f5b4b5cf8-nvzjr\" (UID: \"aca3b4b2-ca5d-4712-9b80-d9d2a41380e6\") " pod="openshift-monitoring/metrics-server-7f5b4b5cf8-nvzjr"
Jul 29 03:56:54.948165 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:54.948144    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/aca3b4b2-ca5d-4712-9b80-d9d2a41380e6-client-ca-bundle\") pod \"metrics-server-7f5b4b5cf8-nvzjr\" (UID: \"aca3b4b2-ca5d-4712-9b80-d9d2a41380e6\") " pod="openshift-monitoring/metrics-server-7f5b4b5cf8-nvzjr"
Jul 29 03:56:54.954377 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:54.954358    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d2bx\" (UniqueName: \"kubernetes.io/projected/aca3b4b2-ca5d-4712-9b80-d9d2a41380e6-kube-api-access-6d2bx\") pod \"metrics-server-7f5b4b5cf8-nvzjr\" (UID: \"aca3b4b2-ca5d-4712-9b80-d9d2a41380e6\") " pod="openshift-monitoring/metrics-server-7f5b4b5cf8-nvzjr"
Jul 29 03:56:55.088767 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:55.088729    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7f5b4b5cf8-nvzjr"
Jul 29 03:56:55.234527 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:55.234496    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7f5b4b5cf8-nvzjr"]
Jul 29 03:56:55.244790 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:56:55.242518    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaca3b4b2_ca5d_4712_9b80_d9d2a41380e6.slice/crio-287fe81853b947248d34886342a4ea776ac8e9f17ed53cb41e8207568ef8a40e WatchSource:0}: Error finding container 287fe81853b947248d34886342a4ea776ac8e9f17ed53cb41e8207568ef8a40e: Status 404 returned error can't find the container with id 287fe81853b947248d34886342a4ea776ac8e9f17ed53cb41e8207568ef8a40e
Jul 29 03:56:55.282800 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:55.282765    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-p2t7j"]
Jul 29 03:56:55.287579 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:55.287555    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-p2t7j"
Jul 29 03:56:55.290299 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:55.290274    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 29 03:56:55.290402 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:55.290281    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-dsrmv\""
Jul 29 03:56:55.294251 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:55.294229    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-p2t7j"]
Jul 29 03:56:55.305655 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:55.305634    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5477ff644-xzdff"]
Jul 29 03:56:55.336507 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:55.336471    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-67568957fb-k2bzw"]
Jul 29 03:56:55.340124 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:55.340102    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67568957fb-k2bzw"
Jul 29 03:56:55.351416 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:55.351365    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x85d4\" (UniqueName: \"kubernetes.io/projected/1caf42ad-2327-496f-8c05-a3059abcf892-kube-api-access-x85d4\") pod \"monitoring-plugin-68b7fc9574-p2t7j\" (UID: \"1caf42ad-2327-496f-8c05-a3059abcf892\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-p2t7j"
Jul 29 03:56:55.351638 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:55.351580    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1caf42ad-2327-496f-8c05-a3059abcf892-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-p2t7j\" (UID: \"1caf42ad-2327-496f-8c05-a3059abcf892\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-p2t7j"
Jul 29 03:56:55.352689 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:55.352670    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67568957fb-k2bzw"]
Jul 29 03:56:55.396895 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:55.396859    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7f5b4b5cf8-nvzjr" event={"ID":"aca3b4b2-ca5d-4712-9b80-d9d2a41380e6","Type":"ContainerStarted","Data":"287fe81853b947248d34886342a4ea776ac8e9f17ed53cb41e8207568ef8a40e"}
Jul 29 03:56:55.398367 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:55.398300    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8686848888-grwsz" event={"ID":"4b2f4957-a89a-4c88-beff-248aca3c6c89","Type":"ContainerStarted","Data":"94e25242923ce20b3a600652502027a39c83b91b37698f34a90f5e3a05e137b5"}
Jul 29 03:56:55.453021 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:55.452960    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e918add3-1a2a-4487-8cec-8bc92196cc4c-console-config\") pod \"console-67568957fb-k2bzw\" (UID: \"e918add3-1a2a-4487-8cec-8bc92196cc4c\") " pod="openshift-console/console-67568957fb-k2bzw"
Jul 29 03:56:55.453021 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:55.453009    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x85d4\" (UniqueName: \"kubernetes.io/projected/1caf42ad-2327-496f-8c05-a3059abcf892-kube-api-access-x85d4\") pod \"monitoring-plugin-68b7fc9574-p2t7j\" (UID: \"1caf42ad-2327-496f-8c05-a3059abcf892\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-p2t7j"
Jul 29 03:56:55.453177 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:55.453028    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e918add3-1a2a-4487-8cec-8bc92196cc4c-console-serving-cert\") pod \"console-67568957fb-k2bzw\" (UID: \"e918add3-1a2a-4487-8cec-8bc92196cc4c\") " pod="openshift-console/console-67568957fb-k2bzw"
Jul 29 03:56:55.453232 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:55.453179    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1caf42ad-2327-496f-8c05-a3059abcf892-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-p2t7j\" (UID: \"1caf42ad-2327-496f-8c05-a3059abcf892\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-p2t7j"
Jul 29 03:56:55.453267 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:55.453253    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e918add3-1a2a-4487-8cec-8bc92196cc4c-service-ca\") pod \"console-67568957fb-k2bzw\" (UID: \"e918add3-1a2a-4487-8cec-8bc92196cc4c\") " pod="openshift-console/console-67568957fb-k2bzw"
Jul 29 03:56:55.453360 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:56:55.453338    2570 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Jul 29 03:56:55.453399 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:55.453370    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppfxj\" (UniqueName: \"kubernetes.io/projected/e918add3-1a2a-4487-8cec-8bc92196cc4c-kube-api-access-ppfxj\") pod \"console-67568957fb-k2bzw\" (UID: \"e918add3-1a2a-4487-8cec-8bc92196cc4c\") " pod="openshift-console/console-67568957fb-k2bzw"
Jul 29 03:56:55.453399 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:56:55.453389    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1caf42ad-2327-496f-8c05-a3059abcf892-monitoring-plugin-cert podName:1caf42ad-2327-496f-8c05-a3059abcf892 nodeName:}" failed. No retries permitted until 2026-07-29 03:56:55.953374851 +0000 UTC m=+80.683505898 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/1caf42ad-2327-496f-8c05-a3059abcf892-monitoring-plugin-cert") pod "monitoring-plugin-68b7fc9574-p2t7j" (UID: "1caf42ad-2327-496f-8c05-a3059abcf892") : secret "monitoring-plugin-cert" not found
Jul 29 03:56:55.453481 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:55.453426    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e918add3-1a2a-4487-8cec-8bc92196cc4c-oauth-serving-cert\") pod \"console-67568957fb-k2bzw\" (UID: \"e918add3-1a2a-4487-8cec-8bc92196cc4c\") " pod="openshift-console/console-67568957fb-k2bzw"
Jul 29 03:56:55.453531 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:55.453514    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e918add3-1a2a-4487-8cec-8bc92196cc4c-trusted-ca-bundle\") pod \"console-67568957fb-k2bzw\" (UID: \"e918add3-1a2a-4487-8cec-8bc92196cc4c\") " pod="openshift-console/console-67568957fb-k2bzw"
Jul 29 03:56:55.453578 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:55.453545    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e918add3-1a2a-4487-8cec-8bc92196cc4c-console-oauth-config\") pod \"console-67568957fb-k2bzw\" (UID: \"e918add3-1a2a-4487-8cec-8bc92196cc4c\") " pod="openshift-console/console-67568957fb-k2bzw"
Jul 29 03:56:55.464278 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:55.464253    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x85d4\" (UniqueName: \"kubernetes.io/projected/1caf42ad-2327-496f-8c05-a3059abcf892-kube-api-access-x85d4\") pod \"monitoring-plugin-68b7fc9574-p2t7j\" (UID: \"1caf42ad-2327-496f-8c05-a3059abcf892\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-p2t7j"
Jul 29 03:56:55.554866 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:55.554810    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e918add3-1a2a-4487-8cec-8bc92196cc4c-service-ca\") pod \"console-67568957fb-k2bzw\" (UID: \"e918add3-1a2a-4487-8cec-8bc92196cc4c\") " pod="openshift-console/console-67568957fb-k2bzw"
Jul 29 03:56:55.555050 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:55.554891    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppfxj\" (UniqueName: \"kubernetes.io/projected/e918add3-1a2a-4487-8cec-8bc92196cc4c-kube-api-access-ppfxj\") pod \"console-67568957fb-k2bzw\" (UID: \"e918add3-1a2a-4487-8cec-8bc92196cc4c\") " pod="openshift-console/console-67568957fb-k2bzw"
Jul 29 03:56:55.555050 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:55.554936    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e918add3-1a2a-4487-8cec-8bc92196cc4c-oauth-serving-cert\") pod \"console-67568957fb-k2bzw\" (UID: \"e918add3-1a2a-4487-8cec-8bc92196cc4c\") " pod="openshift-console/console-67568957fb-k2bzw"
Jul 29 03:56:55.555050 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:55.554981    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e918add3-1a2a-4487-8cec-8bc92196cc4c-trusted-ca-bundle\") pod \"console-67568957fb-k2bzw\" (UID: \"e918add3-1a2a-4487-8cec-8bc92196cc4c\") " pod="openshift-console/console-67568957fb-k2bzw"
Jul 29 03:56:55.555050 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:55.555010    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e918add3-1a2a-4487-8cec-8bc92196cc4c-console-oauth-config\") pod \"console-67568957fb-k2bzw\" (UID: \"e918add3-1a2a-4487-8cec-8bc92196cc4c\") " pod="openshift-console/console-67568957fb-k2bzw"
Jul 29 03:56:55.555255 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:55.555069    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e918add3-1a2a-4487-8cec-8bc92196cc4c-console-config\") pod \"console-67568957fb-k2bzw\" (UID: \"e918add3-1a2a-4487-8cec-8bc92196cc4c\") " pod="openshift-console/console-67568957fb-k2bzw"
Jul 29 03:56:55.555255 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:55.555108    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e918add3-1a2a-4487-8cec-8bc92196cc4c-console-serving-cert\") pod \"console-67568957fb-k2bzw\" (UID: \"e918add3-1a2a-4487-8cec-8bc92196cc4c\") " pod="openshift-console/console-67568957fb-k2bzw"
Jul 29 03:56:55.555620 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:55.555557    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e918add3-1a2a-4487-8cec-8bc92196cc4c-service-ca\") pod \"console-67568957fb-k2bzw\" (UID: \"e918add3-1a2a-4487-8cec-8bc92196cc4c\") " pod="openshift-console/console-67568957fb-k2bzw"
Jul 29 03:56:55.556666 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:55.555989    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e918add3-1a2a-4487-8cec-8bc92196cc4c-oauth-serving-cert\") pod \"console-67568957fb-k2bzw\" (UID: \"e918add3-1a2a-4487-8cec-8bc92196cc4c\") " pod="openshift-console/console-67568957fb-k2bzw"
Jul 29 03:56:55.556666 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:55.556219    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e918add3-1a2a-4487-8cec-8bc92196cc4c-console-config\") pod \"console-67568957fb-k2bzw\" (UID: \"e918add3-1a2a-4487-8cec-8bc92196cc4c\") " pod="openshift-console/console-67568957fb-k2bzw"
Jul 29 03:56:55.556666 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:55.556624    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e918add3-1a2a-4487-8cec-8bc92196cc4c-trusted-ca-bundle\") pod \"console-67568957fb-k2bzw\" (UID: \"e918add3-1a2a-4487-8cec-8bc92196cc4c\") " pod="openshift-console/console-67568957fb-k2bzw"
Jul 29 03:56:55.558332 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:55.558280    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e918add3-1a2a-4487-8cec-8bc92196cc4c-console-oauth-config\") pod \"console-67568957fb-k2bzw\" (UID: \"e918add3-1a2a-4487-8cec-8bc92196cc4c\") " pod="openshift-console/console-67568957fb-k2bzw"
Jul 29 03:56:55.558446 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:55.558420    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e918add3-1a2a-4487-8cec-8bc92196cc4c-console-serving-cert\") pod \"console-67568957fb-k2bzw\" (UID: \"e918add3-1a2a-4487-8cec-8bc92196cc4c\") " pod="openshift-console/console-67568957fb-k2bzw"
Jul 29 03:56:55.564844 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:55.564767    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppfxj\" (UniqueName: \"kubernetes.io/projected/e918add3-1a2a-4487-8cec-8bc92196cc4c-kube-api-access-ppfxj\") pod \"console-67568957fb-k2bzw\" (UID: \"e918add3-1a2a-4487-8cec-8bc92196cc4c\") " pod="openshift-console/console-67568957fb-k2bzw"
Jul 29 03:56:55.650889 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:55.650855    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67568957fb-k2bzw"
Jul 29 03:56:55.958400 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:55.958367    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1caf42ad-2327-496f-8c05-a3059abcf892-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-p2t7j\" (UID: \"1caf42ad-2327-496f-8c05-a3059abcf892\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-p2t7j"
Jul 29 03:56:55.962448 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:55.962421    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1caf42ad-2327-496f-8c05-a3059abcf892-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-p2t7j\" (UID: \"1caf42ad-2327-496f-8c05-a3059abcf892\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-p2t7j"
Jul 29 03:56:55.992877 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:55.992846    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67568957fb-k2bzw"]
Jul 29 03:56:55.996118 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:56:55.996091    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode918add3_1a2a_4487_8cec_8bc92196cc4c.slice/crio-73a8c2c7814c234d0b088e36d5799ea19078361e7cc9d4357b562a970b1b90e5 WatchSource:0}: Error finding container 73a8c2c7814c234d0b088e36d5799ea19078361e7cc9d4357b562a970b1b90e5: Status 404 returned error can't find the container with id 73a8c2c7814c234d0b088e36d5799ea19078361e7cc9d4357b562a970b1b90e5
Jul 29 03:56:56.202623 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.202341    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-p2t7j"
Jul 29 03:56:56.401361 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.401312    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-p2t7j"]
Jul 29 03:56:56.401745 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:56:56.401450    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1caf42ad_2327_496f_8c05_a3059abcf892.slice/crio-ad2859114969e33e4317382b1157b044c9d561561b7b82ed97081b6dcf9cee35 WatchSource:0}: Error finding container ad2859114969e33e4317382b1157b044c9d561561b7b82ed97081b6dcf9cee35: Status 404 returned error can't find the container with id ad2859114969e33e4317382b1157b044c9d561561b7b82ed97081b6dcf9cee35
Jul 29 03:56:56.410728 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.410692    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1416a1fd-24a7-4d3c-93e7-5d8198df9cac","Type":"ContainerStarted","Data":"85e00d969f38c9b5d613f0d577d7b788d1ded97d9a35b62ffa9e2c32608e7cb6"}
Jul 29 03:56:56.410891 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.410798    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1416a1fd-24a7-4d3c-93e7-5d8198df9cac","Type":"ContainerStarted","Data":"c1b9d01551f533678bcf1a6be9e5ff03768b518f7e24df4977dc110a045bd2a3"}
Jul 29 03:56:56.410891 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.410885    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1416a1fd-24a7-4d3c-93e7-5d8198df9cac","Type":"ContainerStarted","Data":"bd1e4ef347a7839d1888324e612f7276ead3afa6bde5629d745d432a8f3c7f4f"}
Jul 29 03:56:56.411002 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.410947    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1416a1fd-24a7-4d3c-93e7-5d8198df9cac","Type":"ContainerStarted","Data":"45da4c141b981e4fb8f615b256ed45347079a3abfcc0100e252a6f524c4cdd9a"}
Jul 29 03:56:56.411054 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.411005    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1416a1fd-24a7-4d3c-93e7-5d8198df9cac","Type":"ContainerStarted","Data":"a70073cf320bb3b9284266267471cb84969669521820c3565183bc875f440f76"}
Jul 29 03:56:56.414274 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.413512    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67568957fb-k2bzw" event={"ID":"e918add3-1a2a-4487-8cec-8bc92196cc4c","Type":"ContainerStarted","Data":"3d55d62427aea640de54d4d531432309a0034e02b8af4998974badbf61829f2e"}
Jul 29 03:56:56.414274 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.413558    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67568957fb-k2bzw" event={"ID":"e918add3-1a2a-4487-8cec-8bc92196cc4c","Type":"ContainerStarted","Data":"73a8c2c7814c234d0b088e36d5799ea19078361e7cc9d4357b562a970b1b90e5"}
Jul 29 03:56:56.444153 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.444099    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-67568957fb-k2bzw" podStartSLOduration=1.444079224 podStartE2EDuration="1.444079224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 03:56:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:56:56.44268093 +0000 UTC m=+81.172812020" watchObservedRunningTime="2026-07-29 03:56:56.444079224 +0000 UTC m=+81.174210294"
Jul 29 03:56:56.814672 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.812275    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 29 03:56:56.817107 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.817083    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 03:56:56.821308 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.821274    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-b2dxt\""
Jul 29 03:56:56.821876 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.821856    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-dmnfb0dohlt11\""
Jul 29 03:56:56.821982 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.821917    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 29 03:56:56.822154 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.822134    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 29 03:56:56.822276 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.822218    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 29 03:56:56.822357 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.822335    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 29 03:56:56.822445 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.822421    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 29 03:56:56.822813 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.822655    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 29 03:56:56.823151 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.823126    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 29 03:56:56.823240 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.823191    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 29 03:56:56.823297 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.823192    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 29 03:56:56.823888 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.823872    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 29 03:56:56.826384 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.826367    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 29 03:56:56.826465 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.826434    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 29 03:56:56.831805 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.831684    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 29 03:56:56.832496 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.832479    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 29 03:56:56.869995 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.869960    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3cd902a6-d01f-469e-85f0-0d6fd7ff42c6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3cd902a6-d01f-469e-85f0-0d6fd7ff42c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 03:56:56.870147 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.870004    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4nzp\" (UniqueName: \"kubernetes.io/projected/3cd902a6-d01f-469e-85f0-0d6fd7ff42c6-kube-api-access-v4nzp\") pod \"prometheus-k8s-0\" (UID: \"3cd902a6-d01f-469e-85f0-0d6fd7ff42c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 03:56:56.870147 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.870037    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3cd902a6-d01f-469e-85f0-0d6fd7ff42c6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3cd902a6-d01f-469e-85f0-0d6fd7ff42c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 03:56:56.870147 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.870066    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3cd902a6-d01f-469e-85f0-0d6fd7ff42c6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3cd902a6-d01f-469e-85f0-0d6fd7ff42c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 03:56:56.870147 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.870094    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3cd902a6-d01f-469e-85f0-0d6fd7ff42c6-config-out\") pod \"prometheus-k8s-0\" (UID: \"3cd902a6-d01f-469e-85f0-0d6fd7ff42c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 03:56:56.870147 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.870117    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3cd902a6-d01f-469e-85f0-0d6fd7ff42c6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3cd902a6-d01f-469e-85f0-0d6fd7ff42c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 03:56:56.870147 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.870141    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3cd902a6-d01f-469e-85f0-0d6fd7ff42c6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3cd902a6-d01f-469e-85f0-0d6fd7ff42c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 03:56:56.870435 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.870173    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3cd902a6-d01f-469e-85f0-0d6fd7ff42c6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3cd902a6-d01f-469e-85f0-0d6fd7ff42c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 03:56:56.870435 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.870193    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3cd902a6-d01f-469e-85f0-0d6fd7ff42c6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3cd902a6-d01f-469e-85f0-0d6fd7ff42c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 03:56:56.870435 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.870218    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3cd902a6-d01f-469e-85f0-0d6fd7ff42c6-web-config\") pod \"prometheus-k8s-0\" (UID: \"3cd902a6-d01f-469e-85f0-0d6fd7ff42c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 03:56:56.870435 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.870248    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3cd902a6-d01f-469e-85f0-0d6fd7ff42c6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3cd902a6-d01f-469e-85f0-0d6fd7ff42c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 03:56:56.870435 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.870284    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3cd902a6-d01f-469e-85f0-0d6fd7ff42c6-config\") pod \"prometheus-k8s-0\" (UID: \"3cd902a6-d01f-469e-85f0-0d6fd7ff42c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 03:56:56.870435 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.870310    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3cd902a6-d01f-469e-85f0-0d6fd7ff42c6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3cd902a6-d01f-469e-85f0-0d6fd7ff42c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 03:56:56.870435 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.870337    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3cd902a6-d01f-469e-85f0-0d6fd7ff42c6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3cd902a6-d01f-469e-85f0-0d6fd7ff42c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 03:56:56.870435 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.870362    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3cd902a6-d01f-469e-85f0-0d6fd7ff42c6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3cd902a6-d01f-469e-85f0-0d6fd7ff42c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 03:56:56.870435 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.870397    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3cd902a6-d01f-469e-85f0-0d6fd7ff42c6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3cd902a6-d01f-469e-85f0-0d6fd7ff42c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 03:56:56.870435 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.870423    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3cd902a6-d01f-469e-85f0-0d6fd7ff42c6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3cd902a6-d01f-469e-85f0-0d6fd7ff42c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 03:56:56.870912 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.870443    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3cd902a6-d01f-469e-85f0-0d6fd7ff42c6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3cd902a6-d01f-469e-85f0-0d6fd7ff42c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 03:56:56.870912 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.870472    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3cd902a6-d01f-469e-85f0-0d6fd7ff42c6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3cd902a6-d01f-469e-85f0-0d6fd7ff42c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 03:56:56.870912 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.870514    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3cd902a6-d01f-469e-85f0-0d6fd7ff42c6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3cd902a6-d01f-469e-85f0-0d6fd7ff42c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 03:56:56.924748 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.924716    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 29 03:56:56.971571 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.971537    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3cd902a6-d01f-469e-85f0-0d6fd7ff42c6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3cd902a6-d01f-469e-85f0-0d6fd7ff42c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 03:56:56.971743 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.971597    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3cd902a6-d01f-469e-85f0-0d6fd7ff42c6-config\") pod \"prometheus-k8s-0\" (UID: \"3cd902a6-d01f-469e-85f0-0d6fd7ff42c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 03:56:56.971743 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.971630    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3cd902a6-d01f-469e-85f0-0d6fd7ff42c6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3cd902a6-d01f-469e-85f0-0d6fd7ff42c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 03:56:56.971743 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.971672    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3cd902a6-d01f-469e-85f0-0d6fd7ff42c6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3cd902a6-d01f-469e-85f0-0d6fd7ff42c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 03:56:56.971743 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.971699    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3cd902a6-d01f-469e-85f0-0d6fd7ff42c6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3cd902a6-d01f-469e-85f0-0d6fd7ff42c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 03:56:56.971743 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.971736    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3cd902a6-d01f-469e-85f0-0d6fd7ff42c6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3cd902a6-d01f-469e-85f0-0d6fd7ff42c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 03:56:56.976601 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.971762    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3cd902a6-d01f-469e-85f0-0d6fd7ff42c6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3cd902a6-d01f-469e-85f0-0d6fd7ff42c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 03:56:56.976601 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.971784    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3cd902a6-d01f-469e-85f0-0d6fd7ff42c6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3cd902a6-d01f-469e-85f0-0d6fd7ff42c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 03:56:56.976601 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.971812    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3cd902a6-d01f-469e-85f0-0d6fd7ff42c6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3cd902a6-d01f-469e-85f0-0d6fd7ff42c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 03:56:56.976601 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.971873    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3cd902a6-d01f-469e-85f0-0d6fd7ff42c6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3cd902a6-d01f-469e-85f0-0d6fd7ff42c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 03:56:56.976601 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.971924    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3cd902a6-d01f-469e-85f0-0d6fd7ff42c6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3cd902a6-d01f-469e-85f0-0d6fd7ff42c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 03:56:56.976601 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.971946    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4nzp\" (UniqueName: \"kubernetes.io/projected/3cd902a6-d01f-469e-85f0-0d6fd7ff42c6-kube-api-access-v4nzp\") pod \"prometheus-k8s-0\" (UID: \"3cd902a6-d01f-469e-85f0-0d6fd7ff42c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 03:56:56.976601 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.971971    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3cd902a6-d01f-469e-85f0-0d6fd7ff42c6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3cd902a6-d01f-469e-85f0-0d6fd7ff42c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 03:56:56.976601 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.972006    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3cd902a6-d01f-469e-85f0-0d6fd7ff42c6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3cd902a6-d01f-469e-85f0-0d6fd7ff42c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 03:56:56.976601 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.972033    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3cd902a6-d01f-469e-85f0-0d6fd7ff42c6-config-out\") pod \"prometheus-k8s-0\" (UID: \"3cd902a6-d01f-469e-85f0-0d6fd7ff42c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 03:56:56.976601 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.972056    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3cd902a6-d01f-469e-85f0-0d6fd7ff42c6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3cd902a6-d01f-469e-85f0-0d6fd7ff42c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 03:56:56.976601 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.972560    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3cd902a6-d01f-469e-85f0-0d6fd7ff42c6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3cd902a6-d01f-469e-85f0-0d6fd7ff42c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 03:56:56.976601 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.973417    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3cd902a6-d01f-469e-85f0-0d6fd7ff42c6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3cd902a6-d01f-469e-85f0-0d6fd7ff42c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 03:56:56.976601 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.973470    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3cd902a6-d01f-469e-85f0-0d6fd7ff42c6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3cd902a6-d01f-469e-85f0-0d6fd7ff42c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 03:56:56.976601 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.973498    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3cd902a6-d01f-469e-85f0-0d6fd7ff42c6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3cd902a6-d01f-469e-85f0-0d6fd7ff42c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 03:56:56.976601 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.973531    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3cd902a6-d01f-469e-85f0-0d6fd7ff42c6-web-config\") pod \"prometheus-k8s-0\" (UID: \"3cd902a6-d01f-469e-85f0-0d6fd7ff42c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 03:56:56.976601 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.975866    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3cd902a6-d01f-469e-85f0-0d6fd7ff42c6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3cd902a6-d01f-469e-85f0-0d6fd7ff42c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 03:56:56.976601 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.976253    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3cd902a6-d01f-469e-85f0-0d6fd7ff42c6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3cd902a6-d01f-469e-85f0-0d6fd7ff42c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 03:56:56.977435 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.976292    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3cd902a6-d01f-469e-85f0-0d6fd7ff42c6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3cd902a6-d01f-469e-85f0-0d6fd7ff42c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 03:56:56.977435 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.976745    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3cd902a6-d01f-469e-85f0-0d6fd7ff42c6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3cd902a6-d01f-469e-85f0-0d6fd7ff42c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 03:56:56.977435 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.976886    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3cd902a6-d01f-469e-85f0-0d6fd7ff42c6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3cd902a6-d01f-469e-85f0-0d6fd7ff42c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 03:56:56.977435 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.977180    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3cd902a6-d01f-469e-85f0-0d6fd7ff42c6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3cd902a6-d01f-469e-85f0-0d6fd7ff42c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 03:56:56.977435 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.977222    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3cd902a6-d01f-469e-85f0-0d6fd7ff42c6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3cd902a6-d01f-469e-85f0-0d6fd7ff42c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 03:56:56.977435 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.977272    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3cd902a6-d01f-469e-85f0-0d6fd7ff42c6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3cd902a6-d01f-469e-85f0-0d6fd7ff42c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 03:56:56.977720 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.977509    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3cd902a6-d01f-469e-85f0-0d6fd7ff42c6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3cd902a6-d01f-469e-85f0-0d6fd7ff42c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 03:56:56.977720 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.977630    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/3cd902a6-d01f-469e-85f0-0d6fd7ff42c6-config\") pod \"prometheus-k8s-0\" (UID: \"3cd902a6-d01f-469e-85f0-0d6fd7ff42c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 03:56:56.977936 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.977915    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3cd902a6-d01f-469e-85f0-0d6fd7ff42c6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3cd902a6-d01f-469e-85f0-0d6fd7ff42c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 03:56:56.978344 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.978289    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3cd902a6-d01f-469e-85f0-0d6fd7ff42c6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3cd902a6-d01f-469e-85f0-0d6fd7ff42c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 03:56:56.978509 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.978483    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3cd902a6-d01f-469e-85f0-0d6fd7ff42c6-config-out\") pod \"prometheus-k8s-0\" (UID: \"3cd902a6-d01f-469e-85f0-0d6fd7ff42c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 03:56:56.978583 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.978542    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3cd902a6-d01f-469e-85f0-0d6fd7ff42c6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3cd902a6-d01f-469e-85f0-0d6fd7ff42c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 03:56:56.978721 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.978663    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3cd902a6-d01f-469e-85f0-0d6fd7ff42c6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3cd902a6-d01f-469e-85f0-0d6fd7ff42c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 03:56:56.979183 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.979144    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3cd902a6-d01f-469e-85f0-0d6fd7ff42c6-web-config\") pod \"prometheus-k8s-0\" (UID: \"3cd902a6-d01f-469e-85f0-0d6fd7ff42c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 03:56:56.980059 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.980039    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3cd902a6-d01f-469e-85f0-0d6fd7ff42c6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3cd902a6-d01f-469e-85f0-0d6fd7ff42c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 03:56:56.980158 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.980085    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3cd902a6-d01f-469e-85f0-0d6fd7ff42c6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3cd902a6-d01f-469e-85f0-0d6fd7ff42c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 03:56:56.984781 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:56.984759    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4nzp\" (UniqueName: \"kubernetes.io/projected/3cd902a6-d01f-469e-85f0-0d6fd7ff42c6-kube-api-access-v4nzp\") pod \"prometheus-k8s-0\" (UID: \"3cd902a6-d01f-469e-85f0-0d6fd7ff42c6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 03:56:57.132046 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:57.132001    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 03:56:57.422885 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:57.422776    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-p2t7j" event={"ID":"1caf42ad-2327-496f-8c05-a3059abcf892","Type":"ContainerStarted","Data":"ad2859114969e33e4317382b1157b044c9d561561b7b82ed97081b6dcf9cee35"}
Jul 29 03:56:57.740353 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:57.740226    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 29 03:56:57.742044 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:56:57.742012    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cd902a6_d01f_469e_85f0_0d6fd7ff42c6.slice/crio-2957e67d4a4e27a81fa3636fee58d5ce05aee5e7d4462608917c6a91b0b08d08 WatchSource:0}: Error finding container 2957e67d4a4e27a81fa3636fee58d5ce05aee5e7d4462608917c6a91b0b08d08: Status 404 returned error can't find the container with id 2957e67d4a4e27a81fa3636fee58d5ce05aee5e7d4462608917c6a91b0b08d08
Jul 29 03:56:58.431369 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:58.431338    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7f5b4b5cf8-nvzjr" event={"ID":"aca3b4b2-ca5d-4712-9b80-d9d2a41380e6","Type":"ContainerStarted","Data":"630de28ad4911d39e551c74cf4a73e99fae5176bdf69776c5fa4d99aba175722"}
Jul 29 03:56:58.433862 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:58.433838    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8686848888-grwsz" event={"ID":"4b2f4957-a89a-4c88-beff-248aca3c6c89","Type":"ContainerStarted","Data":"2c0cac4baa74f96f499805f9b5187de8eb964868e5e02569062b5b92cc257fa8"}
Jul 29 03:56:58.433953 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:58.433909    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8686848888-grwsz" event={"ID":"4b2f4957-a89a-4c88-beff-248aca3c6c89","Type":"ContainerStarted","Data":"fd96eecc6eb46f69a3d803040d111a267d9a0403051e8e59216f5fd98b4f948e"}
Jul 29 03:56:58.434010 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:58.433953    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8686848888-grwsz" event={"ID":"4b2f4957-a89a-4c88-beff-248aca3c6c89","Type":"ContainerStarted","Data":"f41d1bf4d705fd1bc728f7ba918c48d8f7e6834efd6bfea05985bfbd8226d6b7"}
Jul 29 03:56:58.435415 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:58.435391    2570 generic.go:356] "Generic (PLEG): container finished" podID="3cd902a6-d01f-469e-85f0-0d6fd7ff42c6" containerID="e1a31d62ed136cc178765682c2cd4a30175bc42f76ee457d062cddca3026c847" exitCode=0
Jul 29 03:56:58.435479 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:58.435463    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3cd902a6-d01f-469e-85f0-0d6fd7ff42c6","Type":"ContainerDied","Data":"e1a31d62ed136cc178765682c2cd4a30175bc42f76ee457d062cddca3026c847"}
Jul 29 03:56:58.435536 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:58.435523    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3cd902a6-d01f-469e-85f0-0d6fd7ff42c6","Type":"ContainerStarted","Data":"2957e67d4a4e27a81fa3636fee58d5ce05aee5e7d4462608917c6a91b0b08d08"}
Jul 29 03:56:58.449532 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:58.449483    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7f5b4b5cf8-nvzjr" podStartSLOduration=2.123011377 podStartE2EDuration="4.44946483s" totalImagesPullingTime="2.326453453s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 03:56:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:56:58.449284124 +0000 UTC m=+83.179415214" watchObservedRunningTime="2026-07-29 03:56:58.44946483 +0000 UTC m=+83.179595901"
Jul 29 03:56:59.440769 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:59.440714    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-p2t7j" event={"ID":"1caf42ad-2327-496f-8c05-a3059abcf892","Type":"ContainerStarted","Data":"f6793adb7999169fe878d18e2e68bf107219f04f0d21dcb8c5e8648f78d09b5a"}
Jul 29 03:56:59.441220 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:59.440844    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-p2t7j"
Jul 29 03:56:59.444600 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:59.444569    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1416a1fd-24a7-4d3c-93e7-5d8198df9cac","Type":"ContainerStarted","Data":"a727e7539c07f627d93d32ca6ce9686bda5a529fe7b12d6183cd82f1d783dd68"}
Jul 29 03:56:59.446137 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:59.446116    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-p2t7j"
Jul 29 03:56:59.447972 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:59.447952    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8686848888-grwsz" event={"ID":"4b2f4957-a89a-4c88-beff-248aca3c6c89","Type":"ContainerStarted","Data":"492d97f5b3b91047b8ad473411c940378f4d2efbb93b8288dbd5a995ce42c21a"}
Jul 29 03:56:59.448057 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:59.448003    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8686848888-grwsz" event={"ID":"4b2f4957-a89a-4c88-beff-248aca3c6c89","Type":"ContainerStarted","Data":"1c57d009e91eb25025f76fcc972a605e044cb8d7c2342c0a7d4e7cd2c2b31e99"}
Jul 29 03:56:59.448057 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:59.448031    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8686848888-grwsz" event={"ID":"4b2f4957-a89a-4c88-beff-248aca3c6c89","Type":"ContainerStarted","Data":"597328f0189f08bb2f85ae07b2d199d45915836611dab67fb4c4098a3d7d788d"}
Jul 29 03:56:59.458293 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:59.458257    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-p2t7j" podStartSLOduration=2.444853111 podStartE2EDuration="4.458247006s" totalImagesPullingTime="2.013393895s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 03:56:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:56:59.456975117 +0000 UTC m=+84.187106188" watchObservedRunningTime="2026-07-29 03:56:59.458247006 +0000 UTC m=+84.188378075"
Jul 29 03:56:59.503025 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:59.502973    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.830080478 podStartE2EDuration="8.502952199s" totalImagesPullingTime="4.657410516s" totalInitContainerRuntime="15.461205ms" isStatefulPod=false podCreationTimestamp="2026-07-29 03:56:51 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:56:59.501416804 +0000 UTC m=+84.231547877" watchObservedRunningTime="2026-07-29 03:56:59.502952199 +0000 UTC m=+84.233083270"
Jul 29 03:56:59.524660 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:56:59.524609    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-8686848888-grwsz" podStartSLOduration=2.279814891 podStartE2EDuration="5.524589964s" totalImagesPullingTime="3.244775073s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 03:56:54 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:56:59.522771262 +0000 UTC m=+84.252902384" watchObservedRunningTime="2026-07-29 03:56:59.524589964 +0000 UTC m=+84.254721036"
Jul 29 03:57:00.453322 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:00.453250    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-8686848888-grwsz"
Jul 29 03:57:02.465457 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:02.465418    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3cd902a6-d01f-469e-85f0-0d6fd7ff42c6","Type":"ContainerStarted","Data":"ea77f734b6d809e9b0d9ccac1d314b5ac5d52c0c988c39aad6eb2e6dacb26a05"}
Jul 29 03:57:02.465846 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:02.465498    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3cd902a6-d01f-469e-85f0-0d6fd7ff42c6","Type":"ContainerStarted","Data":"5a3b9944352001b0bdaf83b09163af5cbe0cfb444ab0e5c23cc72d2ebd0d64a6"}
Jul 29 03:57:02.465846 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:02.465540    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3cd902a6-d01f-469e-85f0-0d6fd7ff42c6","Type":"ContainerStarted","Data":"44349dbbe1220985cd514eb4035df8cbd707cc79f8312da7da0cab9313a5f5c9"}
Jul 29 03:57:02.465846 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:02.465578    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3cd902a6-d01f-469e-85f0-0d6fd7ff42c6","Type":"ContainerStarted","Data":"e4c5eea2c725609e6c1baf4dd375004d1e65adbd6e9212c24f39ec40d3617ea2"}
Jul 29 03:57:02.465846 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:02.465611    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3cd902a6-d01f-469e-85f0-0d6fd7ff42c6","Type":"ContainerStarted","Data":"092e5ea4c369631d0b1d785c7ca4fa26e2a0ac5455d317c38db2fb88c9b09502"}
Jul 29 03:57:02.465846 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:02.465645    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3cd902a6-d01f-469e-85f0-0d6fd7ff42c6","Type":"ContainerStarted","Data":"33ab9aca6f633ef4b44dc3c27ebd95967e36c9bf2f301ead2a8133582026509f"}
Jul 29 03:57:02.495000 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:02.494929    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.299445998 podStartE2EDuration="6.494911417s" totalImagesPullingTime="3.16841119s" totalInitContainerRuntime="27.054229ms" isStatefulPod=false podCreationTimestamp="2026-07-29 03:56:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:57:02.492564933 +0000 UTC m=+87.222696042" watchObservedRunningTime="2026-07-29 03:57:02.494911417 +0000 UTC m=+87.225042487"
Jul 29 03:57:05.651223 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:05.651190    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-67568957fb-k2bzw"
Jul 29 03:57:05.651745 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:05.651247    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-67568957fb-k2bzw"
Jul 29 03:57:05.655939 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:05.655920    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-67568957fb-k2bzw"
Jul 29 03:57:06.465007 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:06.464978    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-8686848888-grwsz"
Jul 29 03:57:06.487771 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:06.487744    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-67568957fb-k2bzw"
Jul 29 03:57:06.538253 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:06.538214    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-749c48b96f-q7lv6"]
Jul 29 03:57:07.132756 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:07.132721    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 03:57:15.089605 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:15.089574    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7f5b4b5cf8-nvzjr"
Jul 29 03:57:15.089998 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:15.089631    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7f5b4b5cf8-nvzjr"
Jul 29 03:57:19.358520 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:19.358482    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-8222s"
Jul 29 03:57:20.327423 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:20.327375    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5477ff644-xzdff" podUID="3c5a62a3-4ad6-4fbb-a360-7a29be811e12" containerName="console" containerID="cri-o://49d408104cd8475a586318d960ca9b69abd7c472bc9d0a54ea34278cfce53a82" gracePeriod=15
Jul 29 03:57:21.094014 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:21.093990    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5477ff644-xzdff"
Jul 29 03:57:21.193411 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:21.193383    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3c5a62a3-4ad6-4fbb-a360-7a29be811e12-console-config\") pod \"3c5a62a3-4ad6-4fbb-a360-7a29be811e12\" (UID: \"3c5a62a3-4ad6-4fbb-a360-7a29be811e12\") "
Jul 29 03:57:21.193411 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:21.193417    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3c5a62a3-4ad6-4fbb-a360-7a29be811e12-console-serving-cert\") pod \"3c5a62a3-4ad6-4fbb-a360-7a29be811e12\" (UID: \"3c5a62a3-4ad6-4fbb-a360-7a29be811e12\") "
Jul 29 03:57:21.193587 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:21.193437    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3c5a62a3-4ad6-4fbb-a360-7a29be811e12-console-oauth-config\") pod \"3c5a62a3-4ad6-4fbb-a360-7a29be811e12\" (UID: \"3c5a62a3-4ad6-4fbb-a360-7a29be811e12\") "
Jul 29 03:57:21.193587 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:21.193488    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3c5a62a3-4ad6-4fbb-a360-7a29be811e12-service-ca\") pod \"3c5a62a3-4ad6-4fbb-a360-7a29be811e12\" (UID: \"3c5a62a3-4ad6-4fbb-a360-7a29be811e12\") "
Jul 29 03:57:21.193587 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:21.193525    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c5a62a3-4ad6-4fbb-a360-7a29be811e12-trusted-ca-bundle\") pod \"3c5a62a3-4ad6-4fbb-a360-7a29be811e12\" (UID: \"3c5a62a3-4ad6-4fbb-a360-7a29be811e12\") "
Jul 29 03:57:21.193587 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:21.193554    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3c5a62a3-4ad6-4fbb-a360-7a29be811e12-oauth-serving-cert\") pod \"3c5a62a3-4ad6-4fbb-a360-7a29be811e12\" (UID: \"3c5a62a3-4ad6-4fbb-a360-7a29be811e12\") "
Jul 29 03:57:21.193779 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:21.193590    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4s98z\" (UniqueName: \"kubernetes.io/projected/3c5a62a3-4ad6-4fbb-a360-7a29be811e12-kube-api-access-4s98z\") pod \"3c5a62a3-4ad6-4fbb-a360-7a29be811e12\" (UID: \"3c5a62a3-4ad6-4fbb-a360-7a29be811e12\") "
Jul 29 03:57:21.193863 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:21.193766    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c5a62a3-4ad6-4fbb-a360-7a29be811e12-console-config" (OuterVolumeSpecName: "console-config") pod "3c5a62a3-4ad6-4fbb-a360-7a29be811e12" (UID: "3c5a62a3-4ad6-4fbb-a360-7a29be811e12"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 03:57:21.193959 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:21.193928    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c5a62a3-4ad6-4fbb-a360-7a29be811e12-service-ca" (OuterVolumeSpecName: "service-ca") pod "3c5a62a3-4ad6-4fbb-a360-7a29be811e12" (UID: "3c5a62a3-4ad6-4fbb-a360-7a29be811e12"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 03:57:21.194029 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:21.193956    2570 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3c5a62a3-4ad6-4fbb-a360-7a29be811e12-console-config\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 03:57:21.194029 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:21.193984    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c5a62a3-4ad6-4fbb-a360-7a29be811e12-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "3c5a62a3-4ad6-4fbb-a360-7a29be811e12" (UID: "3c5a62a3-4ad6-4fbb-a360-7a29be811e12"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 03:57:21.194120 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:21.194024    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c5a62a3-4ad6-4fbb-a360-7a29be811e12-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "3c5a62a3-4ad6-4fbb-a360-7a29be811e12" (UID: "3c5a62a3-4ad6-4fbb-a360-7a29be811e12"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 03:57:21.195986 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:21.195961    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c5a62a3-4ad6-4fbb-a360-7a29be811e12-kube-api-access-4s98z" (OuterVolumeSpecName: "kube-api-access-4s98z") pod "3c5a62a3-4ad6-4fbb-a360-7a29be811e12" (UID: "3c5a62a3-4ad6-4fbb-a360-7a29be811e12"). InnerVolumeSpecName "kube-api-access-4s98z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 03:57:21.196087 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:21.195996    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c5a62a3-4ad6-4fbb-a360-7a29be811e12-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "3c5a62a3-4ad6-4fbb-a360-7a29be811e12" (UID: "3c5a62a3-4ad6-4fbb-a360-7a29be811e12"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 03:57:21.196132 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:21.196099    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c5a62a3-4ad6-4fbb-a360-7a29be811e12-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "3c5a62a3-4ad6-4fbb-a360-7a29be811e12" (UID: "3c5a62a3-4ad6-4fbb-a360-7a29be811e12"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 03:57:21.295380 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:21.295317    2570 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3c5a62a3-4ad6-4fbb-a360-7a29be811e12-service-ca\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 03:57:21.295380 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:21.295340    2570 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c5a62a3-4ad6-4fbb-a360-7a29be811e12-trusted-ca-bundle\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 03:57:21.295380 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:21.295349    2570 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3c5a62a3-4ad6-4fbb-a360-7a29be811e12-oauth-serving-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 03:57:21.295380 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:21.295358    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4s98z\" (UniqueName: \"kubernetes.io/projected/3c5a62a3-4ad6-4fbb-a360-7a29be811e12-kube-api-access-4s98z\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 03:57:21.295380 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:21.295368    2570 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3c5a62a3-4ad6-4fbb-a360-7a29be811e12-console-serving-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 03:57:21.295380 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:21.295384    2570 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3c5a62a3-4ad6-4fbb-a360-7a29be811e12-console-oauth-config\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 03:57:21.554702 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:21.554609    2570 generic.go:356] "Generic (PLEG): container finished" podID="3c5a62a3-4ad6-4fbb-a360-7a29be811e12" containerID="49d408104cd8475a586318d960ca9b69abd7c472bc9d0a54ea34278cfce53a82" exitCode=0
Jul 29 03:57:21.554702 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:21.554668    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5477ff644-xzdff"
Jul 29 03:57:21.554702 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:21.554679    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5477ff644-xzdff" event={"ID":"3c5a62a3-4ad6-4fbb-a360-7a29be811e12","Type":"ContainerDied","Data":"49d408104cd8475a586318d960ca9b69abd7c472bc9d0a54ea34278cfce53a82"}
Jul 29 03:57:21.554991 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:21.554755    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5477ff644-xzdff" event={"ID":"3c5a62a3-4ad6-4fbb-a360-7a29be811e12","Type":"ContainerDied","Data":"09326713463bbdc2aabe6d8eb4c9f539850d3969140f7ffb1127c90c1305fbf2"}
Jul 29 03:57:21.554991 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:21.554782    2570 scope.go:117] "RemoveContainer" containerID="49d408104cd8475a586318d960ca9b69abd7c472bc9d0a54ea34278cfce53a82"
Jul 29 03:57:21.563780 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:21.563762    2570 scope.go:117] "RemoveContainer" containerID="49d408104cd8475a586318d960ca9b69abd7c472bc9d0a54ea34278cfce53a82"
Jul 29 03:57:21.564050 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:57:21.564030    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49d408104cd8475a586318d960ca9b69abd7c472bc9d0a54ea34278cfce53a82\": container with ID starting with 49d408104cd8475a586318d960ca9b69abd7c472bc9d0a54ea34278cfce53a82 not found: ID does not exist" containerID="49d408104cd8475a586318d960ca9b69abd7c472bc9d0a54ea34278cfce53a82"
Jul 29 03:57:21.564125 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:21.564061    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49d408104cd8475a586318d960ca9b69abd7c472bc9d0a54ea34278cfce53a82"} err="failed to get container status \"49d408104cd8475a586318d960ca9b69abd7c472bc9d0a54ea34278cfce53a82\": rpc error: code = NotFound desc = could not find container \"49d408104cd8475a586318d960ca9b69abd7c472bc9d0a54ea34278cfce53a82\": container with ID starting with 49d408104cd8475a586318d960ca9b69abd7c472bc9d0a54ea34278cfce53a82 not found: ID does not exist"
Jul 29 03:57:21.577342 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:21.577317    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5477ff644-xzdff"]
Jul 29 03:57:21.580977 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:21.580958    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5477ff644-xzdff"]
Jul 29 03:57:21.833326 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:21.833295    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c5a62a3-4ad6-4fbb-a360-7a29be811e12" path="/var/lib/kubelet/pods/3c5a62a3-4ad6-4fbb-a360-7a29be811e12/volumes"
Jul 29 03:57:31.558176 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:31.558134    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-749c48b96f-q7lv6" podUID="99a37005-e877-417e-a9a2-98a9bed210f5" containerName="console" containerID="cri-o://a48568c70da6380b9d300f45d2e5d09b1b3ce1f12e4462869f85d1ae99302940" gracePeriod=15
Jul 29 03:57:31.800303 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:31.800275    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-749c48b96f-q7lv6"
Jul 29 03:57:31.877340 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:31.877316    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/99a37005-e877-417e-a9a2-98a9bed210f5-service-ca\") pod \"99a37005-e877-417e-a9a2-98a9bed210f5\" (UID: \"99a37005-e877-417e-a9a2-98a9bed210f5\") "
Jul 29 03:57:31.877518 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:31.877384    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/99a37005-e877-417e-a9a2-98a9bed210f5-oauth-serving-cert\") pod \"99a37005-e877-417e-a9a2-98a9bed210f5\" (UID: \"99a37005-e877-417e-a9a2-98a9bed210f5\") "
Jul 29 03:57:31.877518 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:31.877424    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g8flp\" (UniqueName: \"kubernetes.io/projected/99a37005-e877-417e-a9a2-98a9bed210f5-kube-api-access-g8flp\") pod \"99a37005-e877-417e-a9a2-98a9bed210f5\" (UID: \"99a37005-e877-417e-a9a2-98a9bed210f5\") "
Jul 29 03:57:31.877518 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:31.877472    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/99a37005-e877-417e-a9a2-98a9bed210f5-console-oauth-config\") pod \"99a37005-e877-417e-a9a2-98a9bed210f5\" (UID: \"99a37005-e877-417e-a9a2-98a9bed210f5\") "
Jul 29 03:57:31.877789 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:31.877757    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99a37005-e877-417e-a9a2-98a9bed210f5-service-ca" (OuterVolumeSpecName: "service-ca") pod "99a37005-e877-417e-a9a2-98a9bed210f5" (UID: "99a37005-e877-417e-a9a2-98a9bed210f5"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 03:57:31.878169 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:31.878137    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99a37005-e877-417e-a9a2-98a9bed210f5-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "99a37005-e877-417e-a9a2-98a9bed210f5" (UID: "99a37005-e877-417e-a9a2-98a9bed210f5"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 03:57:31.880020 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:31.879999    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99a37005-e877-417e-a9a2-98a9bed210f5-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "99a37005-e877-417e-a9a2-98a9bed210f5" (UID: "99a37005-e877-417e-a9a2-98a9bed210f5"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 03:57:31.880314 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:31.880287    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99a37005-e877-417e-a9a2-98a9bed210f5-kube-api-access-g8flp" (OuterVolumeSpecName: "kube-api-access-g8flp") pod "99a37005-e877-417e-a9a2-98a9bed210f5" (UID: "99a37005-e877-417e-a9a2-98a9bed210f5"). InnerVolumeSpecName "kube-api-access-g8flp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 03:57:31.978635 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:31.978602    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/99a37005-e877-417e-a9a2-98a9bed210f5-console-serving-cert\") pod \"99a37005-e877-417e-a9a2-98a9bed210f5\" (UID: \"99a37005-e877-417e-a9a2-98a9bed210f5\") "
Jul 29 03:57:31.978635 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:31.978638    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/99a37005-e877-417e-a9a2-98a9bed210f5-console-config\") pod \"99a37005-e877-417e-a9a2-98a9bed210f5\" (UID: \"99a37005-e877-417e-a9a2-98a9bed210f5\") "
Jul 29 03:57:31.978898 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:31.978779    2570 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/99a37005-e877-417e-a9a2-98a9bed210f5-oauth-serving-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 03:57:31.978898 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:31.978790    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8flp\" (UniqueName: \"kubernetes.io/projected/99a37005-e877-417e-a9a2-98a9bed210f5-kube-api-access-g8flp\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 03:57:31.978898 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:31.978801    2570 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/99a37005-e877-417e-a9a2-98a9bed210f5-console-oauth-config\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 03:57:31.978898 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:31.978810    2570 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/99a37005-e877-417e-a9a2-98a9bed210f5-service-ca\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 03:57:31.979117 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:31.979088    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99a37005-e877-417e-a9a2-98a9bed210f5-console-config" (OuterVolumeSpecName: "console-config") pod "99a37005-e877-417e-a9a2-98a9bed210f5" (UID: "99a37005-e877-417e-a9a2-98a9bed210f5"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 03:57:31.980855 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:31.980833    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99a37005-e877-417e-a9a2-98a9bed210f5-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "99a37005-e877-417e-a9a2-98a9bed210f5" (UID: "99a37005-e877-417e-a9a2-98a9bed210f5"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 03:57:32.079325 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:32.079297    2570 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/99a37005-e877-417e-a9a2-98a9bed210f5-console-serving-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 03:57:32.079325 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:32.079320    2570 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/99a37005-e877-417e-a9a2-98a9bed210f5-console-config\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 03:57:32.604649 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:32.604614    2570 generic.go:356] "Generic (PLEG): container finished" podID="99a37005-e877-417e-a9a2-98a9bed210f5" containerID="a48568c70da6380b9d300f45d2e5d09b1b3ce1f12e4462869f85d1ae99302940" exitCode=0
Jul 29 03:57:32.605035 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:32.604655    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-749c48b96f-q7lv6"
Jul 29 03:57:32.605035 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:32.604690    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-749c48b96f-q7lv6" event={"ID":"99a37005-e877-417e-a9a2-98a9bed210f5","Type":"ContainerDied","Data":"a48568c70da6380b9d300f45d2e5d09b1b3ce1f12e4462869f85d1ae99302940"}
Jul 29 03:57:32.605035 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:32.604739    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-749c48b96f-q7lv6" event={"ID":"99a37005-e877-417e-a9a2-98a9bed210f5","Type":"ContainerDied","Data":"a90de46001955987f51c2d468c40caec191cf27d06d8c158440b2b29ab4d084c"}
Jul 29 03:57:32.605035 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:32.604762    2570 scope.go:117] "RemoveContainer" containerID="a48568c70da6380b9d300f45d2e5d09b1b3ce1f12e4462869f85d1ae99302940"
Jul 29 03:57:32.613437 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:32.613418    2570 scope.go:117] "RemoveContainer" containerID="a48568c70da6380b9d300f45d2e5d09b1b3ce1f12e4462869f85d1ae99302940"
Jul 29 03:57:32.613685 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:57:32.613667    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a48568c70da6380b9d300f45d2e5d09b1b3ce1f12e4462869f85d1ae99302940\": container with ID starting with a48568c70da6380b9d300f45d2e5d09b1b3ce1f12e4462869f85d1ae99302940 not found: ID does not exist" containerID="a48568c70da6380b9d300f45d2e5d09b1b3ce1f12e4462869f85d1ae99302940"
Jul 29 03:57:32.613738 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:32.613694    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a48568c70da6380b9d300f45d2e5d09b1b3ce1f12e4462869f85d1ae99302940"} err="failed to get container status \"a48568c70da6380b9d300f45d2e5d09b1b3ce1f12e4462869f85d1ae99302940\": rpc error: code = NotFound desc = could not find container \"a48568c70da6380b9d300f45d2e5d09b1b3ce1f12e4462869f85d1ae99302940\": container with ID starting with a48568c70da6380b9d300f45d2e5d09b1b3ce1f12e4462869f85d1ae99302940 not found: ID does not exist"
Jul 29 03:57:32.627037 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:32.627013    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-749c48b96f-q7lv6"]
Jul 29 03:57:32.634353 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:32.634326    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-749c48b96f-q7lv6"]
Jul 29 03:57:33.833732 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:33.833700    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99a37005-e877-417e-a9a2-98a9bed210f5" path="/var/lib/kubelet/pods/99a37005-e877-417e-a9a2-98a9bed210f5/volumes"
Jul 29 03:57:35.094457 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:35.094430    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7f5b4b5cf8-nvzjr"
Jul 29 03:57:35.098179 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:35.098158    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7f5b4b5cf8-nvzjr"
Jul 29 03:57:57.133118 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:57.133082    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 03:57:57.153330 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:57.153306    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 03:57:57.725165 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:57:57.725140    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 03:58:19.061089 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:19.061055    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-556dffcb79-p54jn"]
Jul 29 03:58:19.062032 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:19.062009    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c5a62a3-4ad6-4fbb-a360-7a29be811e12" containerName="console"
Jul 29 03:58:19.062167 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:19.062157    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c5a62a3-4ad6-4fbb-a360-7a29be811e12" containerName="console"
Jul 29 03:58:19.062295 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:19.062284    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99a37005-e877-417e-a9a2-98a9bed210f5" containerName="console"
Jul 29 03:58:19.062367 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:19.062358    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="99a37005-e877-417e-a9a2-98a9bed210f5" containerName="console"
Jul 29 03:58:19.062513 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:19.062503    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c5a62a3-4ad6-4fbb-a360-7a29be811e12" containerName="console"
Jul 29 03:58:19.062590 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:19.062582    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="99a37005-e877-417e-a9a2-98a9bed210f5" containerName="console"
Jul 29 03:58:19.066906 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:19.066878    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-556dffcb79-p54jn"
Jul 29 03:58:19.078918 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:19.078893    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-556dffcb79-p54jn"]
Jul 29 03:58:19.182118 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:19.182080    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/643b9665-e798-4dda-a1ea-0474f5651678-console-oauth-config\") pod \"console-556dffcb79-p54jn\" (UID: \"643b9665-e798-4dda-a1ea-0474f5651678\") " pod="openshift-console/console-556dffcb79-p54jn"
Jul 29 03:58:19.182290 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:19.182130    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/643b9665-e798-4dda-a1ea-0474f5651678-console-config\") pod \"console-556dffcb79-p54jn\" (UID: \"643b9665-e798-4dda-a1ea-0474f5651678\") " pod="openshift-console/console-556dffcb79-p54jn"
Jul 29 03:58:19.182290 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:19.182201    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/643b9665-e798-4dda-a1ea-0474f5651678-service-ca\") pod \"console-556dffcb79-p54jn\" (UID: \"643b9665-e798-4dda-a1ea-0474f5651678\") " pod="openshift-console/console-556dffcb79-p54jn"
Jul 29 03:58:19.182360 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:19.182281    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/643b9665-e798-4dda-a1ea-0474f5651678-console-serving-cert\") pod \"console-556dffcb79-p54jn\" (UID: \"643b9665-e798-4dda-a1ea-0474f5651678\") " pod="openshift-console/console-556dffcb79-p54jn"
Jul 29 03:58:19.182360 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:19.182340    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/643b9665-e798-4dda-a1ea-0474f5651678-oauth-serving-cert\") pod \"console-556dffcb79-p54jn\" (UID: \"643b9665-e798-4dda-a1ea-0474f5651678\") " pod="openshift-console/console-556dffcb79-p54jn"
Jul 29 03:58:19.182420 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:19.182366    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5kdr\" (UniqueName: \"kubernetes.io/projected/643b9665-e798-4dda-a1ea-0474f5651678-kube-api-access-w5kdr\") pod \"console-556dffcb79-p54jn\" (UID: \"643b9665-e798-4dda-a1ea-0474f5651678\") " pod="openshift-console/console-556dffcb79-p54jn"
Jul 29 03:58:19.182420 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:19.182397    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/643b9665-e798-4dda-a1ea-0474f5651678-trusted-ca-bundle\") pod \"console-556dffcb79-p54jn\" (UID: \"643b9665-e798-4dda-a1ea-0474f5651678\") " pod="openshift-console/console-556dffcb79-p54jn"
Jul 29 03:58:19.282925 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:19.282889    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/643b9665-e798-4dda-a1ea-0474f5651678-oauth-serving-cert\") pod \"console-556dffcb79-p54jn\" (UID: \"643b9665-e798-4dda-a1ea-0474f5651678\") " pod="openshift-console/console-556dffcb79-p54jn"
Jul 29 03:58:19.282925 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:19.282924    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5kdr\" (UniqueName: \"kubernetes.io/projected/643b9665-e798-4dda-a1ea-0474f5651678-kube-api-access-w5kdr\") pod \"console-556dffcb79-p54jn\" (UID: \"643b9665-e798-4dda-a1ea-0474f5651678\") " pod="openshift-console/console-556dffcb79-p54jn"
Jul 29 03:58:19.283152 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:19.282951    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/643b9665-e798-4dda-a1ea-0474f5651678-trusted-ca-bundle\") pod \"console-556dffcb79-p54jn\" (UID: \"643b9665-e798-4dda-a1ea-0474f5651678\") " pod="openshift-console/console-556dffcb79-p54jn"
Jul 29 03:58:19.283152 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:19.282997    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/643b9665-e798-4dda-a1ea-0474f5651678-console-oauth-config\") pod \"console-556dffcb79-p54jn\" (UID: \"643b9665-e798-4dda-a1ea-0474f5651678\") " pod="openshift-console/console-556dffcb79-p54jn"
Jul 29 03:58:19.283152 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:19.283016    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/643b9665-e798-4dda-a1ea-0474f5651678-console-config\") pod \"console-556dffcb79-p54jn\" (UID: \"643b9665-e798-4dda-a1ea-0474f5651678\") " pod="openshift-console/console-556dffcb79-p54jn"
Jul 29 03:58:19.283152 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:19.283049    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/643b9665-e798-4dda-a1ea-0474f5651678-service-ca\") pod \"console-556dffcb79-p54jn\" (UID: \"643b9665-e798-4dda-a1ea-0474f5651678\") " pod="openshift-console/console-556dffcb79-p54jn"
Jul 29 03:58:19.283152 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:19.283097    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/643b9665-e798-4dda-a1ea-0474f5651678-console-serving-cert\") pod \"console-556dffcb79-p54jn\" (UID: \"643b9665-e798-4dda-a1ea-0474f5651678\") " pod="openshift-console/console-556dffcb79-p54jn"
Jul 29 03:58:19.283721 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:19.283683    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/643b9665-e798-4dda-a1ea-0474f5651678-console-config\") pod \"console-556dffcb79-p54jn\" (UID: \"643b9665-e798-4dda-a1ea-0474f5651678\") " pod="openshift-console/console-556dffcb79-p54jn"
Jul 29 03:58:19.283814 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:19.283689    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/643b9665-e798-4dda-a1ea-0474f5651678-oauth-serving-cert\") pod \"console-556dffcb79-p54jn\" (UID: \"643b9665-e798-4dda-a1ea-0474f5651678\") " pod="openshift-console/console-556dffcb79-p54jn"
Jul 29 03:58:19.283814 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:19.283807    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/643b9665-e798-4dda-a1ea-0474f5651678-service-ca\") pod \"console-556dffcb79-p54jn\" (UID: \"643b9665-e798-4dda-a1ea-0474f5651678\") " pod="openshift-console/console-556dffcb79-p54jn"
Jul 29 03:58:19.284128 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:19.284105    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/643b9665-e798-4dda-a1ea-0474f5651678-trusted-ca-bundle\") pod \"console-556dffcb79-p54jn\" (UID: \"643b9665-e798-4dda-a1ea-0474f5651678\") " pod="openshift-console/console-556dffcb79-p54jn"
Jul 29 03:58:19.285633 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:19.285610    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/643b9665-e798-4dda-a1ea-0474f5651678-console-oauth-config\") pod \"console-556dffcb79-p54jn\" (UID: \"643b9665-e798-4dda-a1ea-0474f5651678\") " pod="openshift-console/console-556dffcb79-p54jn"
Jul 29 03:58:19.285729 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:19.285665    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/643b9665-e798-4dda-a1ea-0474f5651678-console-serving-cert\") pod \"console-556dffcb79-p54jn\" (UID: \"643b9665-e798-4dda-a1ea-0474f5651678\") " pod="openshift-console/console-556dffcb79-p54jn"
Jul 29 03:58:19.292221 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:19.292195    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5kdr\" (UniqueName: \"kubernetes.io/projected/643b9665-e798-4dda-a1ea-0474f5651678-kube-api-access-w5kdr\") pod \"console-556dffcb79-p54jn\" (UID: \"643b9665-e798-4dda-a1ea-0474f5651678\") " pod="openshift-console/console-556dffcb79-p54jn"
Jul 29 03:58:19.377415 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:19.377388    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-556dffcb79-p54jn"
Jul 29 03:58:19.510848 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:19.510790    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-556dffcb79-p54jn"]
Jul 29 03:58:19.513148 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:58:19.513110    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod643b9665_e798_4dda_a1ea_0474f5651678.slice/crio-6ea6e6423e862df57861c097162df495a08218331a6e7f614543ed64946b72f3 WatchSource:0}: Error finding container 6ea6e6423e862df57861c097162df495a08218331a6e7f614543ed64946b72f3: Status 404 returned error can't find the container with id 6ea6e6423e862df57861c097162df495a08218331a6e7f614543ed64946b72f3
Jul 29 03:58:19.798752 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:19.798663    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-556dffcb79-p54jn" event={"ID":"643b9665-e798-4dda-a1ea-0474f5651678","Type":"ContainerStarted","Data":"850a39af4f0e462d6f3ed45f2668791588f48954a79a322c76b135df5223a060"}
Jul 29 03:58:19.798752 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:19.798711    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-556dffcb79-p54jn" event={"ID":"643b9665-e798-4dda-a1ea-0474f5651678","Type":"ContainerStarted","Data":"6ea6e6423e862df57861c097162df495a08218331a6e7f614543ed64946b72f3"}
Jul 29 03:58:19.817114 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:19.817058    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-556dffcb79-p54jn" podStartSLOduration=0.817039841 podStartE2EDuration="817.039841ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 03:58:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:58:19.816650892 +0000 UTC m=+164.546781966" watchObservedRunningTime="2026-07-29 03:58:19.817039841 +0000 UTC m=+164.547170911"
Jul 29 03:58:29.072078 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:29.072005    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-556dffcb79-p54jn"]
Jul 29 03:58:29.099746 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:29.099716    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-598469cf5b-ttwbp"]
Jul 29 03:58:29.102114 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:29.102095    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-598469cf5b-ttwbp"
Jul 29 03:58:29.113624 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:29.113601    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-598469cf5b-ttwbp"]
Jul 29 03:58:29.160532 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:29.160504    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/072b4622-e276-45df-80ae-b870afbbea26-oauth-serving-cert\") pod \"console-598469cf5b-ttwbp\" (UID: \"072b4622-e276-45df-80ae-b870afbbea26\") " pod="openshift-console/console-598469cf5b-ttwbp"
Jul 29 03:58:29.160635 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:29.160541    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/072b4622-e276-45df-80ae-b870afbbea26-service-ca\") pod \"console-598469cf5b-ttwbp\" (UID: \"072b4622-e276-45df-80ae-b870afbbea26\") " pod="openshift-console/console-598469cf5b-ttwbp"
Jul 29 03:58:29.160635 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:29.160561    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/072b4622-e276-45df-80ae-b870afbbea26-trusted-ca-bundle\") pod \"console-598469cf5b-ttwbp\" (UID: \"072b4622-e276-45df-80ae-b870afbbea26\") " pod="openshift-console/console-598469cf5b-ttwbp"
Jul 29 03:58:29.160707 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:29.160631    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf9k7\" (UniqueName: \"kubernetes.io/projected/072b4622-e276-45df-80ae-b870afbbea26-kube-api-access-xf9k7\") pod \"console-598469cf5b-ttwbp\" (UID: \"072b4622-e276-45df-80ae-b870afbbea26\") " pod="openshift-console/console-598469cf5b-ttwbp"
Jul 29 03:58:29.160707 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:29.160654    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/072b4622-e276-45df-80ae-b870afbbea26-console-serving-cert\") pod \"console-598469cf5b-ttwbp\" (UID: \"072b4622-e276-45df-80ae-b870afbbea26\") " pod="openshift-console/console-598469cf5b-ttwbp"
Jul 29 03:58:29.160707 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:29.160695    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/072b4622-e276-45df-80ae-b870afbbea26-console-config\") pod \"console-598469cf5b-ttwbp\" (UID: \"072b4622-e276-45df-80ae-b870afbbea26\") " pod="openshift-console/console-598469cf5b-ttwbp"
Jul 29 03:58:29.160843 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:29.160736    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/072b4622-e276-45df-80ae-b870afbbea26-console-oauth-config\") pod \"console-598469cf5b-ttwbp\" (UID: \"072b4622-e276-45df-80ae-b870afbbea26\") " pod="openshift-console/console-598469cf5b-ttwbp"
Jul 29 03:58:29.261936 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:29.261907    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/072b4622-e276-45df-80ae-b870afbbea26-console-config\") pod \"console-598469cf5b-ttwbp\" (UID: \"072b4622-e276-45df-80ae-b870afbbea26\") " pod="openshift-console/console-598469cf5b-ttwbp"
Jul 29 03:58:29.262060 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:29.261951    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/072b4622-e276-45df-80ae-b870afbbea26-console-oauth-config\") pod \"console-598469cf5b-ttwbp\" (UID: \"072b4622-e276-45df-80ae-b870afbbea26\") " pod="openshift-console/console-598469cf5b-ttwbp"
Jul 29 03:58:29.262060 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:29.261999    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/072b4622-e276-45df-80ae-b870afbbea26-oauth-serving-cert\") pod \"console-598469cf5b-ttwbp\" (UID: \"072b4622-e276-45df-80ae-b870afbbea26\") " pod="openshift-console/console-598469cf5b-ttwbp"
Jul 29 03:58:29.262060 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:29.262037    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/072b4622-e276-45df-80ae-b870afbbea26-service-ca\") pod \"console-598469cf5b-ttwbp\" (UID: \"072b4622-e276-45df-80ae-b870afbbea26\") " pod="openshift-console/console-598469cf5b-ttwbp"
Jul 29 03:58:29.262216 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:29.262068    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/072b4622-e276-45df-80ae-b870afbbea26-trusted-ca-bundle\") pod \"console-598469cf5b-ttwbp\" (UID: \"072b4622-e276-45df-80ae-b870afbbea26\") " pod="openshift-console/console-598469cf5b-ttwbp"
Jul 29 03:58:29.262216 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:29.262121    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf9k7\" (UniqueName: \"kubernetes.io/projected/072b4622-e276-45df-80ae-b870afbbea26-kube-api-access-xf9k7\") pod \"console-598469cf5b-ttwbp\" (UID: \"072b4622-e276-45df-80ae-b870afbbea26\") " pod="openshift-console/console-598469cf5b-ttwbp"
Jul 29 03:58:29.262216 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:29.262149    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/072b4622-e276-45df-80ae-b870afbbea26-console-serving-cert\") pod \"console-598469cf5b-ttwbp\" (UID: \"072b4622-e276-45df-80ae-b870afbbea26\") " pod="openshift-console/console-598469cf5b-ttwbp"
Jul 29 03:58:29.262697 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:29.262667    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/072b4622-e276-45df-80ae-b870afbbea26-oauth-serving-cert\") pod \"console-598469cf5b-ttwbp\" (UID: \"072b4622-e276-45df-80ae-b870afbbea26\") " pod="openshift-console/console-598469cf5b-ttwbp"
Jul 29 03:58:29.262697 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:29.262690    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/072b4622-e276-45df-80ae-b870afbbea26-console-config\") pod \"console-598469cf5b-ttwbp\" (UID: \"072b4622-e276-45df-80ae-b870afbbea26\") " pod="openshift-console/console-598469cf5b-ttwbp"
Jul 29 03:58:29.262867 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:29.262718    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/072b4622-e276-45df-80ae-b870afbbea26-service-ca\") pod \"console-598469cf5b-ttwbp\" (UID: \"072b4622-e276-45df-80ae-b870afbbea26\") " pod="openshift-console/console-598469cf5b-ttwbp"
Jul 29 03:58:29.263127 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:29.263103    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/072b4622-e276-45df-80ae-b870afbbea26-trusted-ca-bundle\") pod \"console-598469cf5b-ttwbp\" (UID: \"072b4622-e276-45df-80ae-b870afbbea26\") " pod="openshift-console/console-598469cf5b-ttwbp"
Jul 29 03:58:29.264442 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:29.264423    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/072b4622-e276-45df-80ae-b870afbbea26-console-oauth-config\") pod \"console-598469cf5b-ttwbp\" (UID: \"072b4622-e276-45df-80ae-b870afbbea26\") " pod="openshift-console/console-598469cf5b-ttwbp"
Jul 29 03:58:29.264611 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:29.264594    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/072b4622-e276-45df-80ae-b870afbbea26-console-serving-cert\") pod \"console-598469cf5b-ttwbp\" (UID: \"072b4622-e276-45df-80ae-b870afbbea26\") " pod="openshift-console/console-598469cf5b-ttwbp"
Jul 29 03:58:29.270042 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:29.270021    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf9k7\" (UniqueName: \"kubernetes.io/projected/072b4622-e276-45df-80ae-b870afbbea26-kube-api-access-xf9k7\") pod \"console-598469cf5b-ttwbp\" (UID: \"072b4622-e276-45df-80ae-b870afbbea26\") " pod="openshift-console/console-598469cf5b-ttwbp"
Jul 29 03:58:29.377874 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:29.377846    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-556dffcb79-p54jn"
Jul 29 03:58:29.411534 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:29.411511    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-598469cf5b-ttwbp"
Jul 29 03:58:29.538396 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:29.538369    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-598469cf5b-ttwbp"]
Jul 29 03:58:29.542469 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:58:29.542437    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod072b4622_e276_45df_80ae_b870afbbea26.slice/crio-915c0093637e54544a968c76bed5ec3125739f19ef9c69befeb8d47798960bd6 WatchSource:0}: Error finding container 915c0093637e54544a968c76bed5ec3125739f19ef9c69befeb8d47798960bd6: Status 404 returned error can't find the container with id 915c0093637e54544a968c76bed5ec3125739f19ef9c69befeb8d47798960bd6
Jul 29 03:58:29.841082 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:29.841050    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-598469cf5b-ttwbp" event={"ID":"072b4622-e276-45df-80ae-b870afbbea26","Type":"ContainerStarted","Data":"f810ba5b7830329a9fb221ddbd547aa1f8d415638cfa57a3522bc4a33d43b3ec"}
Jul 29 03:58:29.841242 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:29.841101    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-598469cf5b-ttwbp" event={"ID":"072b4622-e276-45df-80ae-b870afbbea26","Type":"ContainerStarted","Data":"915c0093637e54544a968c76bed5ec3125739f19ef9c69befeb8d47798960bd6"}
Jul 29 03:58:29.863522 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:29.863474    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-598469cf5b-ttwbp" podStartSLOduration=0.86345863 podStartE2EDuration="863.45863ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 03:58:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:58:29.861204901 +0000 UTC m=+174.591335970" watchObservedRunningTime="2026-07-29 03:58:29.86345863 +0000 UTC m=+174.593589699"
Jul 29 03:58:39.412398 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:39.412358    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-598469cf5b-ttwbp"
Jul 29 03:58:39.412778 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:39.412419    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-598469cf5b-ttwbp"
Jul 29 03:58:39.416872 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:39.416850    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-598469cf5b-ttwbp"
Jul 29 03:58:39.882942 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:39.882911    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-598469cf5b-ttwbp"
Jul 29 03:58:39.929552 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:39.929519    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67568957fb-k2bzw"]
Jul 29 03:58:54.090976 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:54.090906    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-556dffcb79-p54jn" podUID="643b9665-e798-4dda-a1ea-0474f5651678" containerName="console" containerID="cri-o://850a39af4f0e462d6f3ed45f2668791588f48954a79a322c76b135df5223a060" gracePeriod=15
Jul 29 03:58:54.326557 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:54.326534    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-556dffcb79-p54jn"
Jul 29 03:58:54.477277 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:54.477183    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/643b9665-e798-4dda-a1ea-0474f5651678-console-serving-cert\") pod \"643b9665-e798-4dda-a1ea-0474f5651678\" (UID: \"643b9665-e798-4dda-a1ea-0474f5651678\") "
Jul 29 03:58:54.477277 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:54.477221    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/643b9665-e798-4dda-a1ea-0474f5651678-console-oauth-config\") pod \"643b9665-e798-4dda-a1ea-0474f5651678\" (UID: \"643b9665-e798-4dda-a1ea-0474f5651678\") "
Jul 29 03:58:54.477277 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:54.477273    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/643b9665-e798-4dda-a1ea-0474f5651678-oauth-serving-cert\") pod \"643b9665-e798-4dda-a1ea-0474f5651678\" (UID: \"643b9665-e798-4dda-a1ea-0474f5651678\") "
Jul 29 03:58:54.477548 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:54.477386    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/643b9665-e798-4dda-a1ea-0474f5651678-service-ca\") pod \"643b9665-e798-4dda-a1ea-0474f5651678\" (UID: \"643b9665-e798-4dda-a1ea-0474f5651678\") "
Jul 29 03:58:54.477548 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:54.477443    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w5kdr\" (UniqueName: \"kubernetes.io/projected/643b9665-e798-4dda-a1ea-0474f5651678-kube-api-access-w5kdr\") pod \"643b9665-e798-4dda-a1ea-0474f5651678\" (UID: \"643b9665-e798-4dda-a1ea-0474f5651678\") "
Jul 29 03:58:54.477548 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:54.477460    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/643b9665-e798-4dda-a1ea-0474f5651678-trusted-ca-bundle\") pod \"643b9665-e798-4dda-a1ea-0474f5651678\" (UID: \"643b9665-e798-4dda-a1ea-0474f5651678\") "
Jul 29 03:58:54.477548 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:54.477497    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/643b9665-e798-4dda-a1ea-0474f5651678-console-config\") pod \"643b9665-e798-4dda-a1ea-0474f5651678\" (UID: \"643b9665-e798-4dda-a1ea-0474f5651678\") "
Jul 29 03:58:54.477750 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:54.477581    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/643b9665-e798-4dda-a1ea-0474f5651678-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "643b9665-e798-4dda-a1ea-0474f5651678" (UID: "643b9665-e798-4dda-a1ea-0474f5651678"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 03:58:54.477750 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:54.477725    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/643b9665-e798-4dda-a1ea-0474f5651678-service-ca" (OuterVolumeSpecName: "service-ca") pod "643b9665-e798-4dda-a1ea-0474f5651678" (UID: "643b9665-e798-4dda-a1ea-0474f5651678"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 03:58:54.477892 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:54.477784    2570 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/643b9665-e798-4dda-a1ea-0474f5651678-oauth-serving-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 03:58:54.477954 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:54.477924    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/643b9665-e798-4dda-a1ea-0474f5651678-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "643b9665-e798-4dda-a1ea-0474f5651678" (UID: "643b9665-e798-4dda-a1ea-0474f5651678"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 03:58:54.478005 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:54.477965    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/643b9665-e798-4dda-a1ea-0474f5651678-console-config" (OuterVolumeSpecName: "console-config") pod "643b9665-e798-4dda-a1ea-0474f5651678" (UID: "643b9665-e798-4dda-a1ea-0474f5651678"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 03:58:54.479591 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:54.479566    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/643b9665-e798-4dda-a1ea-0474f5651678-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "643b9665-e798-4dda-a1ea-0474f5651678" (UID: "643b9665-e798-4dda-a1ea-0474f5651678"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 03:58:54.479702 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:54.479587    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/643b9665-e798-4dda-a1ea-0474f5651678-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "643b9665-e798-4dda-a1ea-0474f5651678" (UID: "643b9665-e798-4dda-a1ea-0474f5651678"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 03:58:54.479702 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:54.479617    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/643b9665-e798-4dda-a1ea-0474f5651678-kube-api-access-w5kdr" (OuterVolumeSpecName: "kube-api-access-w5kdr") pod "643b9665-e798-4dda-a1ea-0474f5651678" (UID: "643b9665-e798-4dda-a1ea-0474f5651678"). InnerVolumeSpecName "kube-api-access-w5kdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 03:58:54.578979 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:54.578951    2570 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/643b9665-e798-4dda-a1ea-0474f5651678-service-ca\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 03:58:54.578979 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:54.578977    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5kdr\" (UniqueName: \"kubernetes.io/projected/643b9665-e798-4dda-a1ea-0474f5651678-kube-api-access-w5kdr\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 03:58:54.578979 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:54.578987    2570 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/643b9665-e798-4dda-a1ea-0474f5651678-trusted-ca-bundle\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 03:58:54.579175 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:54.578996    2570 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/643b9665-e798-4dda-a1ea-0474f5651678-console-config\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 03:58:54.579175 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:54.579005    2570 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/643b9665-e798-4dda-a1ea-0474f5651678-console-serving-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 03:58:54.579175 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:54.579013    2570 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/643b9665-e798-4dda-a1ea-0474f5651678-console-oauth-config\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 03:58:54.939395 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:54.939365    2570 generic.go:356] "Generic (PLEG): container finished" podID="643b9665-e798-4dda-a1ea-0474f5651678" containerID="850a39af4f0e462d6f3ed45f2668791588f48954a79a322c76b135df5223a060" exitCode=0
Jul 29 03:58:54.939547 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:54.939413    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-556dffcb79-p54jn"
Jul 29 03:58:54.939547 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:54.939426    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-556dffcb79-p54jn" event={"ID":"643b9665-e798-4dda-a1ea-0474f5651678","Type":"ContainerDied","Data":"850a39af4f0e462d6f3ed45f2668791588f48954a79a322c76b135df5223a060"}
Jul 29 03:58:54.939547 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:54.939464    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-556dffcb79-p54jn" event={"ID":"643b9665-e798-4dda-a1ea-0474f5651678","Type":"ContainerDied","Data":"6ea6e6423e862df57861c097162df495a08218331a6e7f614543ed64946b72f3"}
Jul 29 03:58:54.939547 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:54.939483    2570 scope.go:117] "RemoveContainer" containerID="850a39af4f0e462d6f3ed45f2668791588f48954a79a322c76b135df5223a060"
Jul 29 03:58:54.950146 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:54.950126    2570 scope.go:117] "RemoveContainer" containerID="850a39af4f0e462d6f3ed45f2668791588f48954a79a322c76b135df5223a060"
Jul 29 03:58:54.950609 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:58:54.950581    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"850a39af4f0e462d6f3ed45f2668791588f48954a79a322c76b135df5223a060\": container with ID starting with 850a39af4f0e462d6f3ed45f2668791588f48954a79a322c76b135df5223a060 not found: ID does not exist" containerID="850a39af4f0e462d6f3ed45f2668791588f48954a79a322c76b135df5223a060"
Jul 29 03:58:54.950704 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:54.950620    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"850a39af4f0e462d6f3ed45f2668791588f48954a79a322c76b135df5223a060"} err="failed to get container status \"850a39af4f0e462d6f3ed45f2668791588f48954a79a322c76b135df5223a060\": rpc error: code = NotFound desc = could not find container \"850a39af4f0e462d6f3ed45f2668791588f48954a79a322c76b135df5223a060\": container with ID starting with 850a39af4f0e462d6f3ed45f2668791588f48954a79a322c76b135df5223a060 not found: ID does not exist"
Jul 29 03:58:54.960907 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:54.960885    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-556dffcb79-p54jn"]
Jul 29 03:58:54.962187 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:54.962169    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-556dffcb79-p54jn"]
Jul 29 03:58:55.834213 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:58:55.834183    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="643b9665-e798-4dda-a1ea-0474f5651678" path="/var/lib/kubelet/pods/643b9665-e798-4dda-a1ea-0474f5651678/volumes"
Jul 29 03:59:04.950641 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:59:04.950598    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-67568957fb-k2bzw" podUID="e918add3-1a2a-4487-8cec-8bc92196cc4c" containerName="console" containerID="cri-o://3d55d62427aea640de54d4d531432309a0034e02b8af4998974badbf61829f2e" gracePeriod=15
Jul 29 03:59:05.194117 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:59:05.194095    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67568957fb-k2bzw"
Jul 29 03:59:05.370550 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:59:05.370518    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e918add3-1a2a-4487-8cec-8bc92196cc4c-console-serving-cert\") pod \"e918add3-1a2a-4487-8cec-8bc92196cc4c\" (UID: \"e918add3-1a2a-4487-8cec-8bc92196cc4c\") "
Jul 29 03:59:05.370550 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:59:05.370549    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e918add3-1a2a-4487-8cec-8bc92196cc4c-console-oauth-config\") pod \"e918add3-1a2a-4487-8cec-8bc92196cc4c\" (UID: \"e918add3-1a2a-4487-8cec-8bc92196cc4c\") "
Jul 29 03:59:05.370787 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:59:05.370586    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e918add3-1a2a-4487-8cec-8bc92196cc4c-trusted-ca-bundle\") pod \"e918add3-1a2a-4487-8cec-8bc92196cc4c\" (UID: \"e918add3-1a2a-4487-8cec-8bc92196cc4c\") "
Jul 29 03:59:05.370787 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:59:05.370604    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e918add3-1a2a-4487-8cec-8bc92196cc4c-service-ca\") pod \"e918add3-1a2a-4487-8cec-8bc92196cc4c\" (UID: \"e918add3-1a2a-4487-8cec-8bc92196cc4c\") "
Jul 29 03:59:05.370787 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:59:05.370639    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ppfxj\" (UniqueName: \"kubernetes.io/projected/e918add3-1a2a-4487-8cec-8bc92196cc4c-kube-api-access-ppfxj\") pod \"e918add3-1a2a-4487-8cec-8bc92196cc4c\" (UID: \"e918add3-1a2a-4487-8cec-8bc92196cc4c\") "
Jul 29 03:59:05.370787 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:59:05.370673    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e918add3-1a2a-4487-8cec-8bc92196cc4c-oauth-serving-cert\") pod \"e918add3-1a2a-4487-8cec-8bc92196cc4c\" (UID: \"e918add3-1a2a-4487-8cec-8bc92196cc4c\") "
Jul 29 03:59:05.370787 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:59:05.370719    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e918add3-1a2a-4487-8cec-8bc92196cc4c-console-config\") pod \"e918add3-1a2a-4487-8cec-8bc92196cc4c\" (UID: \"e918add3-1a2a-4487-8cec-8bc92196cc4c\") "
Jul 29 03:59:05.371082 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:59:05.371054    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e918add3-1a2a-4487-8cec-8bc92196cc4c-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "e918add3-1a2a-4487-8cec-8bc92196cc4c" (UID: "e918add3-1a2a-4487-8cec-8bc92196cc4c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 03:59:05.371211 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:59:05.371186    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e918add3-1a2a-4487-8cec-8bc92196cc4c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "e918add3-1a2a-4487-8cec-8bc92196cc4c" (UID: "e918add3-1a2a-4487-8cec-8bc92196cc4c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 03:59:05.371211 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:59:05.371191    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e918add3-1a2a-4487-8cec-8bc92196cc4c-service-ca" (OuterVolumeSpecName: "service-ca") pod "e918add3-1a2a-4487-8cec-8bc92196cc4c" (UID: "e918add3-1a2a-4487-8cec-8bc92196cc4c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 03:59:05.371580 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:59:05.371264    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e918add3-1a2a-4487-8cec-8bc92196cc4c-console-config" (OuterVolumeSpecName: "console-config") pod "e918add3-1a2a-4487-8cec-8bc92196cc4c" (UID: "e918add3-1a2a-4487-8cec-8bc92196cc4c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 03:59:05.372993 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:59:05.372959    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e918add3-1a2a-4487-8cec-8bc92196cc4c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "e918add3-1a2a-4487-8cec-8bc92196cc4c" (UID: "e918add3-1a2a-4487-8cec-8bc92196cc4c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 03:59:05.373069 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:59:05.372997    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e918add3-1a2a-4487-8cec-8bc92196cc4c-kube-api-access-ppfxj" (OuterVolumeSpecName: "kube-api-access-ppfxj") pod "e918add3-1a2a-4487-8cec-8bc92196cc4c" (UID: "e918add3-1a2a-4487-8cec-8bc92196cc4c"). InnerVolumeSpecName "kube-api-access-ppfxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 03:59:05.373069 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:59:05.373028    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e918add3-1a2a-4487-8cec-8bc92196cc4c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "e918add3-1a2a-4487-8cec-8bc92196cc4c" (UID: "e918add3-1a2a-4487-8cec-8bc92196cc4c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 03:59:05.472060 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:59:05.472028    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppfxj\" (UniqueName: \"kubernetes.io/projected/e918add3-1a2a-4487-8cec-8bc92196cc4c-kube-api-access-ppfxj\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 03:59:05.472060 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:59:05.472053    2570 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e918add3-1a2a-4487-8cec-8bc92196cc4c-oauth-serving-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 03:59:05.472060 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:59:05.472064    2570 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e918add3-1a2a-4487-8cec-8bc92196cc4c-console-config\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 03:59:05.472251 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:59:05.472073    2570 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e918add3-1a2a-4487-8cec-8bc92196cc4c-console-serving-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 03:59:05.472251 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:59:05.472081    2570 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e918add3-1a2a-4487-8cec-8bc92196cc4c-console-oauth-config\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 03:59:05.472251 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:59:05.472090    2570 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e918add3-1a2a-4487-8cec-8bc92196cc4c-trusted-ca-bundle\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 03:59:05.472251 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:59:05.472097    2570 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e918add3-1a2a-4487-8cec-8bc92196cc4c-service-ca\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 03:59:05.984438 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:59:05.984401    2570 generic.go:356] "Generic (PLEG): container finished" podID="e918add3-1a2a-4487-8cec-8bc92196cc4c" containerID="3d55d62427aea640de54d4d531432309a0034e02b8af4998974badbf61829f2e" exitCode=0
Jul 29 03:59:05.984878 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:59:05.984444    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67568957fb-k2bzw"
Jul 29 03:59:05.984878 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:59:05.984498    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67568957fb-k2bzw" event={"ID":"e918add3-1a2a-4487-8cec-8bc92196cc4c","Type":"ContainerDied","Data":"3d55d62427aea640de54d4d531432309a0034e02b8af4998974badbf61829f2e"}
Jul 29 03:59:05.984878 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:59:05.984542    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67568957fb-k2bzw" event={"ID":"e918add3-1a2a-4487-8cec-8bc92196cc4c","Type":"ContainerDied","Data":"73a8c2c7814c234d0b088e36d5799ea19078361e7cc9d4357b562a970b1b90e5"}
Jul 29 03:59:05.984878 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:59:05.984561    2570 scope.go:117] "RemoveContainer" containerID="3d55d62427aea640de54d4d531432309a0034e02b8af4998974badbf61829f2e"
Jul 29 03:59:05.992994 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:59:05.992976    2570 scope.go:117] "RemoveContainer" containerID="3d55d62427aea640de54d4d531432309a0034e02b8af4998974badbf61829f2e"
Jul 29 03:59:05.993254 ip-10-0-135-138 kubenswrapper[2570]: E0729 03:59:05.993237    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d55d62427aea640de54d4d531432309a0034e02b8af4998974badbf61829f2e\": container with ID starting with 3d55d62427aea640de54d4d531432309a0034e02b8af4998974badbf61829f2e not found: ID does not exist" containerID="3d55d62427aea640de54d4d531432309a0034e02b8af4998974badbf61829f2e"
Jul 29 03:59:05.993305 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:59:05.993262    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d55d62427aea640de54d4d531432309a0034e02b8af4998974badbf61829f2e"} err="failed to get container status \"3d55d62427aea640de54d4d531432309a0034e02b8af4998974badbf61829f2e\": rpc error: code = NotFound desc = could not find container \"3d55d62427aea640de54d4d531432309a0034e02b8af4998974badbf61829f2e\": container with ID starting with 3d55d62427aea640de54d4d531432309a0034e02b8af4998974badbf61829f2e not found: ID does not exist"
Jul 29 03:59:06.000992 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:59:06.000970    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67568957fb-k2bzw"]
Jul 29 03:59:06.004665 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:59:06.004644    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-67568957fb-k2bzw"]
Jul 29 03:59:07.833305 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:59:07.833268    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e918add3-1a2a-4487-8cec-8bc92196cc4c" path="/var/lib/kubelet/pods/e918add3-1a2a-4487-8cec-8bc92196cc4c/volumes"
Jul 29 03:59:39.664912 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:59:39.664881    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-z4ghm"]
Jul 29 03:59:39.665293 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:59:39.665229    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e918add3-1a2a-4487-8cec-8bc92196cc4c" containerName="console"
Jul 29 03:59:39.665293 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:59:39.665241    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="e918add3-1a2a-4487-8cec-8bc92196cc4c" containerName="console"
Jul 29 03:59:39.665293 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:59:39.665259    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="643b9665-e798-4dda-a1ea-0474f5651678" containerName="console"
Jul 29 03:59:39.665293 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:59:39.665265    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="643b9665-e798-4dda-a1ea-0474f5651678" containerName="console"
Jul 29 03:59:39.665418 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:59:39.665318    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="643b9665-e798-4dda-a1ea-0474f5651678" containerName="console"
Jul 29 03:59:39.665418 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:59:39.665328    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="e918add3-1a2a-4487-8cec-8bc92196cc4c" containerName="console"
Jul 29 03:59:39.667992 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:59:39.667975    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-z4ghm"
Jul 29 03:59:39.670751 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:59:39.670721    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 29 03:59:39.671994 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:59:39.671975    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 29 03:59:39.672078 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:59:39.672049    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-pmhzb\""
Jul 29 03:59:39.675669 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:59:39.675639    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-z4ghm"]
Jul 29 03:59:39.732294 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:59:39.732255    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4f53b470-5321-4a14-8c05-370a50324dfa-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-z4ghm\" (UID: \"4f53b470-5321-4a14-8c05-370a50324dfa\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-z4ghm"
Jul 29 03:59:39.732429 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:59:39.732323    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvnzg\" (UniqueName: \"kubernetes.io/projected/4f53b470-5321-4a14-8c05-370a50324dfa-kube-api-access-dvnzg\") pod \"cert-manager-cainjector-559cdd8c87-z4ghm\" (UID: \"4f53b470-5321-4a14-8c05-370a50324dfa\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-z4ghm"
Jul 29 03:59:39.833464 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:59:39.833441    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvnzg\" (UniqueName: \"kubernetes.io/projected/4f53b470-5321-4a14-8c05-370a50324dfa-kube-api-access-dvnzg\") pod \"cert-manager-cainjector-559cdd8c87-z4ghm\" (UID: \"4f53b470-5321-4a14-8c05-370a50324dfa\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-z4ghm"
Jul 29 03:59:39.833570 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:59:39.833494    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4f53b470-5321-4a14-8c05-370a50324dfa-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-z4ghm\" (UID: \"4f53b470-5321-4a14-8c05-370a50324dfa\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-z4ghm"
Jul 29 03:59:39.843238 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:59:39.843217    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4f53b470-5321-4a14-8c05-370a50324dfa-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-z4ghm\" (UID: \"4f53b470-5321-4a14-8c05-370a50324dfa\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-z4ghm"
Jul 29 03:59:39.843437 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:59:39.843414    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvnzg\" (UniqueName: \"kubernetes.io/projected/4f53b470-5321-4a14-8c05-370a50324dfa-kube-api-access-dvnzg\") pod \"cert-manager-cainjector-559cdd8c87-z4ghm\" (UID: \"4f53b470-5321-4a14-8c05-370a50324dfa\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-z4ghm"
Jul 29 03:59:39.995147 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:59:39.995085    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-z4ghm"
Jul 29 03:59:40.123927 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:59:40.123906    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-z4ghm"]
Jul 29 03:59:40.126569 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:59:40.126545    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4f53b470_5321_4a14_8c05_370a50324dfa.slice/crio-827cb75423d4fb551fd1678945aefae12da76355bdcb3941ce20d9093cb1170a WatchSource:0}: Error finding container 827cb75423d4fb551fd1678945aefae12da76355bdcb3941ce20d9093cb1170a: Status 404 returned error can't find the container with id 827cb75423d4fb551fd1678945aefae12da76355bdcb3941ce20d9093cb1170a
Jul 29 03:59:41.128144 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:59:41.128104    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-z4ghm" event={"ID":"4f53b470-5321-4a14-8c05-370a50324dfa","Type":"ContainerStarted","Data":"827cb75423d4fb551fd1678945aefae12da76355bdcb3941ce20d9093cb1170a"}
Jul 29 03:59:43.137658 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:59:43.137619    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-z4ghm" event={"ID":"4f53b470-5321-4a14-8c05-370a50324dfa","Type":"ContainerStarted","Data":"aa3a2a78ce3116c77219e849e25cb458904bccbdc860fe4cc183d7fc1bd46086"}
Jul 29 03:59:43.155243 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:59:43.155186    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-z4ghm" podStartSLOduration=1.30958914 podStartE2EDuration="4.155173142s" totalImagesPullingTime="2.845584002s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 03:59:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:59:43.152940985 +0000 UTC m=+247.883072067" watchObservedRunningTime="2026-07-29 03:59:43.155173142 +0000 UTC m=+247.885304212"
Jul 29 03:59:46.427714 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:59:46.427681    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-wpvzs"]
Jul 29 03:59:46.430153 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:59:46.430137    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-wpvzs"
Jul 29 03:59:46.432874 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:59:46.432857    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-7xf5s\""
Jul 29 03:59:46.439703 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:59:46.439683    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-wpvzs"]
Jul 29 03:59:46.492835 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:59:46.492795    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpj6t\" (UniqueName: \"kubernetes.io/projected/bf8b3e43-7246-4a2d-9635-d633016524aa-kube-api-access-kpj6t\") pod \"cert-manager-55b4d954d5-wpvzs\" (UID: \"bf8b3e43-7246-4a2d-9635-d633016524aa\") " pod="cert-manager/cert-manager-55b4d954d5-wpvzs"
Jul 29 03:59:46.492950 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:59:46.492845    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bf8b3e43-7246-4a2d-9635-d633016524aa-bound-sa-token\") pod \"cert-manager-55b4d954d5-wpvzs\" (UID: \"bf8b3e43-7246-4a2d-9635-d633016524aa\") " pod="cert-manager/cert-manager-55b4d954d5-wpvzs"
Jul 29 03:59:46.593717 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:59:46.593685    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpj6t\" (UniqueName: \"kubernetes.io/projected/bf8b3e43-7246-4a2d-9635-d633016524aa-kube-api-access-kpj6t\") pod \"cert-manager-55b4d954d5-wpvzs\" (UID: \"bf8b3e43-7246-4a2d-9635-d633016524aa\") " pod="cert-manager/cert-manager-55b4d954d5-wpvzs"
Jul 29 03:59:46.593883 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:59:46.593725    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bf8b3e43-7246-4a2d-9635-d633016524aa-bound-sa-token\") pod \"cert-manager-55b4d954d5-wpvzs\" (UID: \"bf8b3e43-7246-4a2d-9635-d633016524aa\") " pod="cert-manager/cert-manager-55b4d954d5-wpvzs"
Jul 29 03:59:46.602142 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:59:46.602116    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bf8b3e43-7246-4a2d-9635-d633016524aa-bound-sa-token\") pod \"cert-manager-55b4d954d5-wpvzs\" (UID: \"bf8b3e43-7246-4a2d-9635-d633016524aa\") " pod="cert-manager/cert-manager-55b4d954d5-wpvzs"
Jul 29 03:59:46.602415 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:59:46.602390    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpj6t\" (UniqueName: \"kubernetes.io/projected/bf8b3e43-7246-4a2d-9635-d633016524aa-kube-api-access-kpj6t\") pod \"cert-manager-55b4d954d5-wpvzs\" (UID: \"bf8b3e43-7246-4a2d-9635-d633016524aa\") " pod="cert-manager/cert-manager-55b4d954d5-wpvzs"
Jul 29 03:59:46.739173 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:59:46.739114    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-wpvzs"
Jul 29 03:59:46.867797 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:59:46.867752    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-wpvzs"]
Jul 29 03:59:46.869833 ip-10-0-135-138 kubenswrapper[2570]: W0729 03:59:46.869787    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbf8b3e43_7246_4a2d_9635_d633016524aa.slice/crio-587c22344bf53432fb123c36067dfd2b4759fa25716cb68211122614f5860d48 WatchSource:0}: Error finding container 587c22344bf53432fb123c36067dfd2b4759fa25716cb68211122614f5860d48: Status 404 returned error can't find the container with id 587c22344bf53432fb123c36067dfd2b4759fa25716cb68211122614f5860d48
Jul 29 03:59:47.154896 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:59:47.154864    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-wpvzs" event={"ID":"bf8b3e43-7246-4a2d-9635-d633016524aa","Type":"ContainerStarted","Data":"e418f11192bfee0f62b8c6d0bf81ccfa71daec099ae6c94b9a8ef6c8b9e69041"}
Jul 29 03:59:47.155050 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:59:47.154909    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-wpvzs" event={"ID":"bf8b3e43-7246-4a2d-9635-d633016524aa","Type":"ContainerStarted","Data":"587c22344bf53432fb123c36067dfd2b4759fa25716cb68211122614f5860d48"}
Jul 29 03:59:47.171561 ip-10-0-135-138 kubenswrapper[2570]: I0729 03:59:47.171515    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-wpvzs" podStartSLOduration=1.171502139 podStartE2EDuration="1.171502139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 03:59:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:59:47.170274592 +0000 UTC m=+251.900405663" watchObservedRunningTime="2026-07-29 03:59:47.171502139 +0000 UTC m=+251.901633208"
Jul 29 04:00:05.362165 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:05.362082    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-86cc57cc46-qjlpt"]
Jul 29 04:00:05.365526 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:05.365509    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-86cc57cc46-qjlpt"
Jul 29 04:00:05.368373 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:05.368354    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 29 04:00:05.368373 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:05.368364    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 29 04:00:05.369844 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:05.369788    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 29 04:00:05.369945 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:05.369788    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 29 04:00:05.369945 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:05.369796    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 29 04:00:05.369945 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:05.369870    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-xhj24\""
Jul 29 04:00:05.374871 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:05.374854    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-86cc57cc46-qjlpt"]
Jul 29 04:00:05.562740 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:05.562700    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a9290a1c-bb9c-4b67-b357-8ff1be3fbeda-cert\") pod \"lws-controller-manager-86cc57cc46-qjlpt\" (UID: \"a9290a1c-bb9c-4b67-b357-8ff1be3fbeda\") " pod="openshift-lws-operator/lws-controller-manager-86cc57cc46-qjlpt"
Jul 29 04:00:05.562920 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:05.562745    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a9290a1c-bb9c-4b67-b357-8ff1be3fbeda-metrics-cert\") pod \"lws-controller-manager-86cc57cc46-qjlpt\" (UID: \"a9290a1c-bb9c-4b67-b357-8ff1be3fbeda\") " pod="openshift-lws-operator/lws-controller-manager-86cc57cc46-qjlpt"
Jul 29 04:00:05.562920 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:05.562765    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58l8g\" (UniqueName: \"kubernetes.io/projected/a9290a1c-bb9c-4b67-b357-8ff1be3fbeda-kube-api-access-58l8g\") pod \"lws-controller-manager-86cc57cc46-qjlpt\" (UID: \"a9290a1c-bb9c-4b67-b357-8ff1be3fbeda\") " pod="openshift-lws-operator/lws-controller-manager-86cc57cc46-qjlpt"
Jul 29 04:00:05.562920 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:05.562894    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a9290a1c-bb9c-4b67-b357-8ff1be3fbeda-manager-config\") pod \"lws-controller-manager-86cc57cc46-qjlpt\" (UID: \"a9290a1c-bb9c-4b67-b357-8ff1be3fbeda\") " pod="openshift-lws-operator/lws-controller-manager-86cc57cc46-qjlpt"
Jul 29 04:00:05.663611 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:05.663534    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a9290a1c-bb9c-4b67-b357-8ff1be3fbeda-cert\") pod \"lws-controller-manager-86cc57cc46-qjlpt\" (UID: \"a9290a1c-bb9c-4b67-b357-8ff1be3fbeda\") " pod="openshift-lws-operator/lws-controller-manager-86cc57cc46-qjlpt"
Jul 29 04:00:05.663611 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:05.663575    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a9290a1c-bb9c-4b67-b357-8ff1be3fbeda-metrics-cert\") pod \"lws-controller-manager-86cc57cc46-qjlpt\" (UID: \"a9290a1c-bb9c-4b67-b357-8ff1be3fbeda\") " pod="openshift-lws-operator/lws-controller-manager-86cc57cc46-qjlpt"
Jul 29 04:00:05.663611 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:05.663593    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58l8g\" (UniqueName: \"kubernetes.io/projected/a9290a1c-bb9c-4b67-b357-8ff1be3fbeda-kube-api-access-58l8g\") pod \"lws-controller-manager-86cc57cc46-qjlpt\" (UID: \"a9290a1c-bb9c-4b67-b357-8ff1be3fbeda\") " pod="openshift-lws-operator/lws-controller-manager-86cc57cc46-qjlpt"
Jul 29 04:00:05.663916 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:05.663742    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a9290a1c-bb9c-4b67-b357-8ff1be3fbeda-manager-config\") pod \"lws-controller-manager-86cc57cc46-qjlpt\" (UID: \"a9290a1c-bb9c-4b67-b357-8ff1be3fbeda\") " pod="openshift-lws-operator/lws-controller-manager-86cc57cc46-qjlpt"
Jul 29 04:00:05.664362 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:05.664341    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a9290a1c-bb9c-4b67-b357-8ff1be3fbeda-manager-config\") pod \"lws-controller-manager-86cc57cc46-qjlpt\" (UID: \"a9290a1c-bb9c-4b67-b357-8ff1be3fbeda\") " pod="openshift-lws-operator/lws-controller-manager-86cc57cc46-qjlpt"
Jul 29 04:00:05.666243 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:05.666215    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a9290a1c-bb9c-4b67-b357-8ff1be3fbeda-cert\") pod \"lws-controller-manager-86cc57cc46-qjlpt\" (UID: \"a9290a1c-bb9c-4b67-b357-8ff1be3fbeda\") " pod="openshift-lws-operator/lws-controller-manager-86cc57cc46-qjlpt"
Jul 29 04:00:05.666243 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:05.666236    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a9290a1c-bb9c-4b67-b357-8ff1be3fbeda-metrics-cert\") pod \"lws-controller-manager-86cc57cc46-qjlpt\" (UID: \"a9290a1c-bb9c-4b67-b357-8ff1be3fbeda\") " pod="openshift-lws-operator/lws-controller-manager-86cc57cc46-qjlpt"
Jul 29 04:00:05.672808 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:05.672780    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58l8g\" (UniqueName: \"kubernetes.io/projected/a9290a1c-bb9c-4b67-b357-8ff1be3fbeda-kube-api-access-58l8g\") pod \"lws-controller-manager-86cc57cc46-qjlpt\" (UID: \"a9290a1c-bb9c-4b67-b357-8ff1be3fbeda\") " pod="openshift-lws-operator/lws-controller-manager-86cc57cc46-qjlpt"
Jul 29 04:00:05.675521 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:05.675498    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-86cc57cc46-qjlpt"
Jul 29 04:00:05.820284 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:05.820262    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-86cc57cc46-qjlpt"]
Jul 29 04:00:05.822519 ip-10-0-135-138 kubenswrapper[2570]: W0729 04:00:05.822489    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9290a1c_bb9c_4b67_b357_8ff1be3fbeda.slice/crio-ab9057d729af27983ec0e969c64e3f7460d50a1cdf0595968028e814972a2703 WatchSource:0}: Error finding container ab9057d729af27983ec0e969c64e3f7460d50a1cdf0595968028e814972a2703: Status 404 returned error can't find the container with id ab9057d729af27983ec0e969c64e3f7460d50a1cdf0595968028e814972a2703
Jul 29 04:00:06.231294 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:06.231264    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-86cc57cc46-qjlpt" event={"ID":"a9290a1c-bb9c-4b67-b357-8ff1be3fbeda","Type":"ContainerStarted","Data":"ab9057d729af27983ec0e969c64e3f7460d50a1cdf0595968028e814972a2703"}
Jul 29 04:00:09.700782 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:09.700746    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d56969f4b-mdm9z"]
Jul 29 04:00:09.710219 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:09.710094    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-d56969f4b-mdm9z"
Jul 29 04:00:09.713252 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:09.713212    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 29 04:00:09.713252 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:09.713227    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-npvqn\""
Jul 29 04:00:09.713526 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:09.713263    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 29 04:00:09.713593 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:09.713576    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 29 04:00:09.713677 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:09.713662    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 29 04:00:09.718273 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:09.718250    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d56969f4b-mdm9z"]
Jul 29 04:00:09.796690 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:09.796658    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8e7db7d0-36b3-460e-8127-910806de4c83-tmp\") pod \"opendatahub-operator-controller-manager-d56969f4b-mdm9z\" (UID: \"8e7db7d0-36b3-460e-8127-910806de4c83\") " pod="opendatahub/opendatahub-operator-controller-manager-d56969f4b-mdm9z"
Jul 29 04:00:09.796690 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:09.796703    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8e7db7d0-36b3-460e-8127-910806de4c83-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d56969f4b-mdm9z\" (UID: \"8e7db7d0-36b3-460e-8127-910806de4c83\") " pod="opendatahub/opendatahub-operator-controller-manager-d56969f4b-mdm9z"
Jul 29 04:00:09.796982 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:09.796737    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8e7db7d0-36b3-460e-8127-910806de4c83-manifests\") pod \"opendatahub-operator-controller-manager-d56969f4b-mdm9z\" (UID: \"8e7db7d0-36b3-460e-8127-910806de4c83\") " pod="opendatahub/opendatahub-operator-controller-manager-d56969f4b-mdm9z"
Jul 29 04:00:09.796982 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:09.796764    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8e7db7d0-36b3-460e-8127-910806de4c83-webhook-cert\") pod \"opendatahub-operator-controller-manager-d56969f4b-mdm9z\" (UID: \"8e7db7d0-36b3-460e-8127-910806de4c83\") " pod="opendatahub/opendatahub-operator-controller-manager-d56969f4b-mdm9z"
Jul 29 04:00:09.796982 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:09.796797    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq5pj\" (UniqueName: \"kubernetes.io/projected/8e7db7d0-36b3-460e-8127-910806de4c83-kube-api-access-mq5pj\") pod \"opendatahub-operator-controller-manager-d56969f4b-mdm9z\" (UID: \"8e7db7d0-36b3-460e-8127-910806de4c83\") " pod="opendatahub/opendatahub-operator-controller-manager-d56969f4b-mdm9z"
Jul 29 04:00:09.897875 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:09.897811    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq5pj\" (UniqueName: \"kubernetes.io/projected/8e7db7d0-36b3-460e-8127-910806de4c83-kube-api-access-mq5pj\") pod \"opendatahub-operator-controller-manager-d56969f4b-mdm9z\" (UID: \"8e7db7d0-36b3-460e-8127-910806de4c83\") " pod="opendatahub/opendatahub-operator-controller-manager-d56969f4b-mdm9z"
Jul 29 04:00:09.898056 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:09.897985    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8e7db7d0-36b3-460e-8127-910806de4c83-tmp\") pod \"opendatahub-operator-controller-manager-d56969f4b-mdm9z\" (UID: \"8e7db7d0-36b3-460e-8127-910806de4c83\") " pod="opendatahub/opendatahub-operator-controller-manager-d56969f4b-mdm9z"
Jul 29 04:00:09.898056 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:09.898027    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8e7db7d0-36b3-460e-8127-910806de4c83-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d56969f4b-mdm9z\" (UID: \"8e7db7d0-36b3-460e-8127-910806de4c83\") " pod="opendatahub/opendatahub-operator-controller-manager-d56969f4b-mdm9z"
Jul 29 04:00:09.898178 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:09.898059    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8e7db7d0-36b3-460e-8127-910806de4c83-manifests\") pod \"opendatahub-operator-controller-manager-d56969f4b-mdm9z\" (UID: \"8e7db7d0-36b3-460e-8127-910806de4c83\") " pod="opendatahub/opendatahub-operator-controller-manager-d56969f4b-mdm9z"
Jul 29 04:00:09.898178 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:09.898079    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8e7db7d0-36b3-460e-8127-910806de4c83-webhook-cert\") pod \"opendatahub-operator-controller-manager-d56969f4b-mdm9z\" (UID: \"8e7db7d0-36b3-460e-8127-910806de4c83\") " pod="opendatahub/opendatahub-operator-controller-manager-d56969f4b-mdm9z"
Jul 29 04:00:09.898476 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:09.898443    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8e7db7d0-36b3-460e-8127-910806de4c83-manifests\") pod \"opendatahub-operator-controller-manager-d56969f4b-mdm9z\" (UID: \"8e7db7d0-36b3-460e-8127-910806de4c83\") " pod="opendatahub/opendatahub-operator-controller-manager-d56969f4b-mdm9z"
Jul 29 04:00:09.900644 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:09.900618    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8e7db7d0-36b3-460e-8127-910806de4c83-tmp\") pod \"opendatahub-operator-controller-manager-d56969f4b-mdm9z\" (UID: \"8e7db7d0-36b3-460e-8127-910806de4c83\") " pod="opendatahub/opendatahub-operator-controller-manager-d56969f4b-mdm9z"
Jul 29 04:00:09.900952 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:09.900930    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8e7db7d0-36b3-460e-8127-910806de4c83-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d56969f4b-mdm9z\" (UID: \"8e7db7d0-36b3-460e-8127-910806de4c83\") " pod="opendatahub/opendatahub-operator-controller-manager-d56969f4b-mdm9z"
Jul 29 04:00:09.901093 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:09.901074    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8e7db7d0-36b3-460e-8127-910806de4c83-webhook-cert\") pod \"opendatahub-operator-controller-manager-d56969f4b-mdm9z\" (UID: \"8e7db7d0-36b3-460e-8127-910806de4c83\") " pod="opendatahub/opendatahub-operator-controller-manager-d56969f4b-mdm9z"
Jul 29 04:00:09.906624 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:09.906600    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq5pj\" (UniqueName: \"kubernetes.io/projected/8e7db7d0-36b3-460e-8127-910806de4c83-kube-api-access-mq5pj\") pod \"opendatahub-operator-controller-manager-d56969f4b-mdm9z\" (UID: \"8e7db7d0-36b3-460e-8127-910806de4c83\") " pod="opendatahub/opendatahub-operator-controller-manager-d56969f4b-mdm9z"
Jul 29 04:00:10.025511 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:10.025427    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-d56969f4b-mdm9z"
Jul 29 04:00:10.460060 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:10.460038    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d56969f4b-mdm9z"]
Jul 29 04:00:10.462460 ip-10-0-135-138 kubenswrapper[2570]: W0729 04:00:10.462429    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e7db7d0_36b3_460e_8127_910806de4c83.slice/crio-a8534ef7ba99c6e0d2f2fcaaf84cde2332102d169b146c37cfe1b302201dc800 WatchSource:0}: Error finding container a8534ef7ba99c6e0d2f2fcaaf84cde2332102d169b146c37cfe1b302201dc800: Status 404 returned error can't find the container with id a8534ef7ba99c6e0d2f2fcaaf84cde2332102d169b146c37cfe1b302201dc800
Jul 29 04:00:11.259836 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:11.259764    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-86cc57cc46-qjlpt" event={"ID":"a9290a1c-bb9c-4b67-b357-8ff1be3fbeda","Type":"ContainerStarted","Data":"f75e3e8261fb60e62605a13e90341d69931ff56928a3d637f1d489fc5fc5f935"}
Jul 29 04:00:11.260337 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:11.260224    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-86cc57cc46-qjlpt"
Jul 29 04:00:11.261719 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:11.261639    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d56969f4b-mdm9z" event={"ID":"8e7db7d0-36b3-460e-8127-910806de4c83","Type":"ContainerStarted","Data":"a8534ef7ba99c6e0d2f2fcaaf84cde2332102d169b146c37cfe1b302201dc800"}
Jul 29 04:00:11.283103 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:11.283041    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-86cc57cc46-qjlpt" podStartSLOduration=1.7098222870000002 podStartE2EDuration="6.283023194s" totalImagesPullingTime="4.573200907s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:00:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:00:11.28170535 +0000 UTC m=+276.011836444" watchObservedRunningTime="2026-07-29 04:00:11.283023194 +0000 UTC m=+276.013154264"
Jul 29 04:00:14.278454 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:14.278418    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d56969f4b-mdm9z" event={"ID":"8e7db7d0-36b3-460e-8127-910806de4c83","Type":"ContainerStarted","Data":"674cec92cc3ecc1f69e6eac36c54040f685a2ce74579c931ae19e70fef0e36f6"}
Jul 29 04:00:15.284668 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:15.284634    2570 generic.go:356] "Generic (PLEG): container finished" podID="8e7db7d0-36b3-460e-8127-910806de4c83" containerID="674cec92cc3ecc1f69e6eac36c54040f685a2ce74579c931ae19e70fef0e36f6" exitCode=0
Jul 29 04:00:15.285072 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:15.284723    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d56969f4b-mdm9z" event={"ID":"8e7db7d0-36b3-460e-8127-910806de4c83","Type":"ContainerDied","Data":"674cec92cc3ecc1f69e6eac36c54040f685a2ce74579c931ae19e70fef0e36f6"}
Jul 29 04:00:16.291317 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:16.291282    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d56969f4b-mdm9z" event={"ID":"8e7db7d0-36b3-460e-8127-910806de4c83","Type":"ContainerStarted","Data":"56b22852fa274e602ec0c36b04d4bc925ce2a5ad0907640f838539e897a31ba8"}
Jul 29 04:00:16.291736 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:16.291413    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-d56969f4b-mdm9z"
Jul 29 04:00:16.314244 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:16.314192    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-d56969f4b-mdm9z" podStartSLOduration=2.751544621 podStartE2EDuration="7.314177261s" totalImagesPullingTime="3.556721856s" totalInitContainerRuntime="1.005910784s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:00:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:00:16.311744729 +0000 UTC m=+281.041875801" watchObservedRunningTime="2026-07-29 04:00:16.314177261 +0000 UTC m=+281.044308330"
Jul 29 04:00:22.269805 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:22.269774    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-86cc57cc46-qjlpt"
Jul 29 04:00:27.297746 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:27.297713    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-d56969f4b-mdm9z"
Jul 29 04:00:35.667019 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:35.666986    2570 kubelet.go:1648] "Image garbage collection succeeded"
Jul 29 04:00:39.863907 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:39.863879    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-664ccd966c-brch5"]
Jul 29 04:00:39.867728 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:39.867711    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-664ccd966c-brch5"
Jul 29 04:00:39.872133 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:39.872108    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 29 04:00:39.872133 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:39.872129    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 29 04:00:39.873623 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:39.873602    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 29 04:00:39.873711 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:39.873602    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-btzts\""
Jul 29 04:00:39.873711 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:39.873673    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 29 04:00:39.888774 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:39.888746    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-664ccd966c-brch5"]
Jul 29 04:00:39.969992 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:39.969956    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/87caebc4-95a0-4bd4-bd96-884de1f86d3b-tmp\") pod \"kube-auth-proxy-664ccd966c-brch5\" (UID: \"87caebc4-95a0-4bd4-bd96-884de1f86d3b\") " pod="openshift-ingress/kube-auth-proxy-664ccd966c-brch5"
Jul 29 04:00:39.969992 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:39.969996    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wcsd\" (UniqueName: \"kubernetes.io/projected/87caebc4-95a0-4bd4-bd96-884de1f86d3b-kube-api-access-4wcsd\") pod \"kube-auth-proxy-664ccd966c-brch5\" (UID: \"87caebc4-95a0-4bd4-bd96-884de1f86d3b\") " pod="openshift-ingress/kube-auth-proxy-664ccd966c-brch5"
Jul 29 04:00:39.970256 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:39.970189    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87caebc4-95a0-4bd4-bd96-884de1f86d3b-tls-certs\") pod \"kube-auth-proxy-664ccd966c-brch5\" (UID: \"87caebc4-95a0-4bd4-bd96-884de1f86d3b\") " pod="openshift-ingress/kube-auth-proxy-664ccd966c-brch5"
Jul 29 04:00:40.071403 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:40.071357    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87caebc4-95a0-4bd4-bd96-884de1f86d3b-tls-certs\") pod \"kube-auth-proxy-664ccd966c-brch5\" (UID: \"87caebc4-95a0-4bd4-bd96-884de1f86d3b\") " pod="openshift-ingress/kube-auth-proxy-664ccd966c-brch5"
Jul 29 04:00:40.071403 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:40.071418    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/87caebc4-95a0-4bd4-bd96-884de1f86d3b-tmp\") pod \"kube-auth-proxy-664ccd966c-brch5\" (UID: \"87caebc4-95a0-4bd4-bd96-884de1f86d3b\") " pod="openshift-ingress/kube-auth-proxy-664ccd966c-brch5"
Jul 29 04:00:40.071657 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:40.071442    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wcsd\" (UniqueName: \"kubernetes.io/projected/87caebc4-95a0-4bd4-bd96-884de1f86d3b-kube-api-access-4wcsd\") pod \"kube-auth-proxy-664ccd966c-brch5\" (UID: \"87caebc4-95a0-4bd4-bd96-884de1f86d3b\") " pod="openshift-ingress/kube-auth-proxy-664ccd966c-brch5"
Jul 29 04:00:40.073694 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:40.073671    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/87caebc4-95a0-4bd4-bd96-884de1f86d3b-tmp\") pod \"kube-auth-proxy-664ccd966c-brch5\" (UID: \"87caebc4-95a0-4bd4-bd96-884de1f86d3b\") " pod="openshift-ingress/kube-auth-proxy-664ccd966c-brch5"
Jul 29 04:00:40.074012 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:40.073990    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87caebc4-95a0-4bd4-bd96-884de1f86d3b-tls-certs\") pod \"kube-auth-proxy-664ccd966c-brch5\" (UID: \"87caebc4-95a0-4bd4-bd96-884de1f86d3b\") " pod="openshift-ingress/kube-auth-proxy-664ccd966c-brch5"
Jul 29 04:00:40.079533 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:40.079510    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wcsd\" (UniqueName: \"kubernetes.io/projected/87caebc4-95a0-4bd4-bd96-884de1f86d3b-kube-api-access-4wcsd\") pod \"kube-auth-proxy-664ccd966c-brch5\" (UID: \"87caebc4-95a0-4bd4-bd96-884de1f86d3b\") " pod="openshift-ingress/kube-auth-proxy-664ccd966c-brch5"
Jul 29 04:00:40.176675 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:40.176610    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-664ccd966c-brch5"
Jul 29 04:00:40.306530 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:40.306495    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-664ccd966c-brch5"]
Jul 29 04:00:40.309183 ip-10-0-135-138 kubenswrapper[2570]: W0729 04:00:40.309155    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87caebc4_95a0_4bd4_bd96_884de1f86d3b.slice/crio-9d157ad67457f37eab4719beb9831f76d4d2a3f0100ed5305510ead92fd8ad8a WatchSource:0}: Error finding container 9d157ad67457f37eab4719beb9831f76d4d2a3f0100ed5305510ead92fd8ad8a: Status 404 returned error can't find the container with id 9d157ad67457f37eab4719beb9831f76d4d2a3f0100ed5305510ead92fd8ad8a
Jul 29 04:00:40.311238 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:40.311218    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 04:00:40.394548 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:40.394515    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-664ccd966c-brch5" event={"ID":"87caebc4-95a0-4bd4-bd96-884de1f86d3b","Type":"ContainerStarted","Data":"9d157ad67457f37eab4719beb9831f76d4d2a3f0100ed5305510ead92fd8ad8a"}
Jul 29 04:00:43.411457 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:43.411419    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-664ccd966c-brch5" event={"ID":"87caebc4-95a0-4bd4-bd96-884de1f86d3b","Type":"ContainerStarted","Data":"1b5ff32b7dc538c457d005627d6f8c933994e43f36e9c26be78c49f8a8909dfa"}
Jul 29 04:00:43.428425 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:00:43.428378    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-664ccd966c-brch5" podStartSLOduration=1.9134078749999999 podStartE2EDuration="4.428361257s" totalImagesPullingTime="2.514953382s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:00:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:00:43.427115683 +0000 UTC m=+308.157246765" watchObservedRunningTime="2026-07-29 04:00:43.428361257 +0000 UTC m=+308.158492327"
Jul 29 04:02:14.150832 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:14.150792    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6x9lh"]
Jul 29 04:02:14.153267 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:14.153250    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6x9lh"
Jul 29 04:02:14.156651 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:14.156631    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 29 04:02:14.157202 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:14.157184    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 29 04:02:14.157202 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:14.157193    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-pc9k8\""
Jul 29 04:02:14.165813 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:14.165794    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6x9lh"]
Jul 29 04:02:14.273743 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:14.273714    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dllcs\" (UniqueName: \"kubernetes.io/projected/418b9f85-2272-4477-ae6b-196bb94fb85b-kube-api-access-dllcs\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6x9lh\" (UID: \"418b9f85-2272-4477-ae6b-196bb94fb85b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6x9lh"
Jul 29 04:02:14.273909 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:14.273758    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/418b9f85-2272-4477-ae6b-196bb94fb85b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6x9lh\" (UID: \"418b9f85-2272-4477-ae6b-196bb94fb85b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6x9lh"
Jul 29 04:02:14.375216 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:14.375181    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dllcs\" (UniqueName: \"kubernetes.io/projected/418b9f85-2272-4477-ae6b-196bb94fb85b-kube-api-access-dllcs\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6x9lh\" (UID: \"418b9f85-2272-4477-ae6b-196bb94fb85b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6x9lh"
Jul 29 04:02:14.375385 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:14.375228    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/418b9f85-2272-4477-ae6b-196bb94fb85b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6x9lh\" (UID: \"418b9f85-2272-4477-ae6b-196bb94fb85b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6x9lh"
Jul 29 04:02:14.375616 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:14.375599    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/418b9f85-2272-4477-ae6b-196bb94fb85b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6x9lh\" (UID: \"418b9f85-2272-4477-ae6b-196bb94fb85b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6x9lh"
Jul 29 04:02:14.388247 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:14.388225    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dllcs\" (UniqueName: \"kubernetes.io/projected/418b9f85-2272-4477-ae6b-196bb94fb85b-kube-api-access-dllcs\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6x9lh\" (UID: \"418b9f85-2272-4477-ae6b-196bb94fb85b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6x9lh"
Jul 29 04:02:14.463754 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:14.463678    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6x9lh"
Jul 29 04:02:14.594018 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:14.593996    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6x9lh"]
Jul 29 04:02:14.598870 ip-10-0-135-138 kubenswrapper[2570]: W0729 04:02:14.596410    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod418b9f85_2272_4477_ae6b_196bb94fb85b.slice/crio-f3adb0a6844429dc48b9e331c34bd8a09861f1c43669398992c6f2c4a24c9f95 WatchSource:0}: Error finding container f3adb0a6844429dc48b9e331c34bd8a09861f1c43669398992c6f2c4a24c9f95: Status 404 returned error can't find the container with id f3adb0a6844429dc48b9e331c34bd8a09861f1c43669398992c6f2c4a24c9f95
Jul 29 04:02:14.812959 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:14.812870    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6x9lh" event={"ID":"418b9f85-2272-4477-ae6b-196bb94fb85b","Type":"ContainerStarted","Data":"f3adb0a6844429dc48b9e331c34bd8a09861f1c43669398992c6f2c4a24c9f95"}
Jul 29 04:02:19.843619 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:19.843533    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6x9lh" event={"ID":"418b9f85-2272-4477-ae6b-196bb94fb85b","Type":"ContainerStarted","Data":"6c942bf3813dd120cae4b256db87f5712ecf2ef5651c2bb41619d14bd4202b62"}
Jul 29 04:02:19.843619 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:19.843606    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6x9lh"
Jul 29 04:02:19.865261 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:19.865222    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6x9lh" podStartSLOduration=0.878437193 podStartE2EDuration="5.86520635s" totalImagesPullingTime="4.986769157s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:02:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:02:19.862895353 +0000 UTC m=+404.593026446" watchObservedRunningTime="2026-07-29 04:02:19.86520635 +0000 UTC m=+404.595337420"
Jul 29 04:02:30.850374 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:30.850341    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6x9lh"
Jul 29 04:02:32.554315 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:32.554280    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6x9lh"]
Jul 29 04:02:32.554870 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:32.554520    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6x9lh" podUID="418b9f85-2272-4477-ae6b-196bb94fb85b" containerName="manager" containerID="cri-o://6c942bf3813dd120cae4b256db87f5712ecf2ef5651c2bb41619d14bd4202b62" gracePeriod=2
Jul 29 04:02:32.565321 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:32.565295    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6x9lh"]
Jul 29 04:02:32.574576 ip-10-0-135-138 kubenswrapper[2570]: E0729 04:02:32.574517    2570 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"418b9f85-2272-4477-ae6b-196bb94fb85b\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-6x9lh\": pods \"kuadrant-operator-controller-manager-8684c9ff49-6x9lh\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6x9lh"
Jul 29 04:02:32.580774 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:32.580751    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p5www"]
Jul 29 04:02:32.581182 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:32.581166    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="418b9f85-2272-4477-ae6b-196bb94fb85b" containerName="manager"
Jul 29 04:02:32.581241 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:32.581186    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="418b9f85-2272-4477-ae6b-196bb94fb85b" containerName="manager"
Jul 29 04:02:32.581275 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:32.581259    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="418b9f85-2272-4477-ae6b-196bb94fb85b" containerName="manager"
Jul 29 04:02:32.591683 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:32.591663    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p5www"
Jul 29 04:02:32.594366 ip-10-0-135-138 kubenswrapper[2570]: E0729 04:02:32.594338    2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6x9lh\" is forbidden: User \"system:node:ip-10-0-135-138.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-138.ec2.internal' and this object" podUID="418b9f85-2272-4477-ae6b-196bb94fb85b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6x9lh"
Jul 29 04:02:32.598449 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:32.598425    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p5www"]
Jul 29 04:02:32.748382 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:32.748356    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9hnn\" (UniqueName: \"kubernetes.io/projected/e325693f-db11-4d81-8ddc-3b3d4c5c35a2-kube-api-access-d9hnn\") pod \"kuadrant-operator-controller-manager-8684c9ff49-p5www\" (UID: \"e325693f-db11-4d81-8ddc-3b3d4c5c35a2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p5www"
Jul 29 04:02:32.748505 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:32.748389    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e325693f-db11-4d81-8ddc-3b3d4c5c35a2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-p5www\" (UID: \"e325693f-db11-4d81-8ddc-3b3d4c5c35a2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p5www"
Jul 29 04:02:32.793198 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:32.793179    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6x9lh"
Jul 29 04:02:32.795771 ip-10-0-135-138 kubenswrapper[2570]: E0729 04:02:32.795744    2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6x9lh\" is forbidden: User \"system:node:ip-10-0-135-138.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-138.ec2.internal' and this object" podUID="418b9f85-2272-4477-ae6b-196bb94fb85b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6x9lh"
Jul 29 04:02:32.849094 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:32.849067    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e325693f-db11-4d81-8ddc-3b3d4c5c35a2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-p5www\" (UID: \"e325693f-db11-4d81-8ddc-3b3d4c5c35a2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p5www"
Jul 29 04:02:32.849195 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:32.849181    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9hnn\" (UniqueName: \"kubernetes.io/projected/e325693f-db11-4d81-8ddc-3b3d4c5c35a2-kube-api-access-d9hnn\") pod \"kuadrant-operator-controller-manager-8684c9ff49-p5www\" (UID: \"e325693f-db11-4d81-8ddc-3b3d4c5c35a2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p5www"
Jul 29 04:02:32.849420 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:32.849402    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e325693f-db11-4d81-8ddc-3b3d4c5c35a2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-p5www\" (UID: \"e325693f-db11-4d81-8ddc-3b3d4c5c35a2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p5www"
Jul 29 04:02:32.863133 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:32.863109    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9hnn\" (UniqueName: \"kubernetes.io/projected/e325693f-db11-4d81-8ddc-3b3d4c5c35a2-kube-api-access-d9hnn\") pod \"kuadrant-operator-controller-manager-8684c9ff49-p5www\" (UID: \"e325693f-db11-4d81-8ddc-3b3d4c5c35a2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p5www"
Jul 29 04:02:32.899809 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:32.899786    2570 generic.go:356] "Generic (PLEG): container finished" podID="418b9f85-2272-4477-ae6b-196bb94fb85b" containerID="6c942bf3813dd120cae4b256db87f5712ecf2ef5651c2bb41619d14bd4202b62" exitCode=0
Jul 29 04:02:32.899926 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:32.899857    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6x9lh"
Jul 29 04:02:32.899926 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:32.899872    2570 scope.go:117] "RemoveContainer" containerID="6c942bf3813dd120cae4b256db87f5712ecf2ef5651c2bb41619d14bd4202b62"
Jul 29 04:02:32.902474 ip-10-0-135-138 kubenswrapper[2570]: E0729 04:02:32.902452    2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6x9lh\" is forbidden: User \"system:node:ip-10-0-135-138.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-138.ec2.internal' and this object" podUID="418b9f85-2272-4477-ae6b-196bb94fb85b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6x9lh"
Jul 29 04:02:32.907994 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:32.907979    2570 scope.go:117] "RemoveContainer" containerID="6c942bf3813dd120cae4b256db87f5712ecf2ef5651c2bb41619d14bd4202b62"
Jul 29 04:02:32.908279 ip-10-0-135-138 kubenswrapper[2570]: E0729 04:02:32.908263    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c942bf3813dd120cae4b256db87f5712ecf2ef5651c2bb41619d14bd4202b62\": container with ID starting with 6c942bf3813dd120cae4b256db87f5712ecf2ef5651c2bb41619d14bd4202b62 not found: ID does not exist" containerID="6c942bf3813dd120cae4b256db87f5712ecf2ef5651c2bb41619d14bd4202b62"
Jul 29 04:02:32.908337 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:32.908288    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c942bf3813dd120cae4b256db87f5712ecf2ef5651c2bb41619d14bd4202b62"} err="failed to get container status \"6c942bf3813dd120cae4b256db87f5712ecf2ef5651c2bb41619d14bd4202b62\": rpc error: code = NotFound desc = could not find container \"6c942bf3813dd120cae4b256db87f5712ecf2ef5651c2bb41619d14bd4202b62\": container with ID starting with 6c942bf3813dd120cae4b256db87f5712ecf2ef5651c2bb41619d14bd4202b62 not found: ID does not exist"
Jul 29 04:02:32.934521 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:32.934499    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p5www"
Jul 29 04:02:32.949730 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:32.949710    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/418b9f85-2272-4477-ae6b-196bb94fb85b-extensions-socket-volume\") pod \"418b9f85-2272-4477-ae6b-196bb94fb85b\" (UID: \"418b9f85-2272-4477-ae6b-196bb94fb85b\") "
Jul 29 04:02:32.949884 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:32.949867    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dllcs\" (UniqueName: \"kubernetes.io/projected/418b9f85-2272-4477-ae6b-196bb94fb85b-kube-api-access-dllcs\") pod \"418b9f85-2272-4477-ae6b-196bb94fb85b\" (UID: \"418b9f85-2272-4477-ae6b-196bb94fb85b\") "
Jul 29 04:02:32.950179 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:32.950159    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/418b9f85-2272-4477-ae6b-196bb94fb85b-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "418b9f85-2272-4477-ae6b-196bb94fb85b" (UID: "418b9f85-2272-4477-ae6b-196bb94fb85b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 04:02:32.951985 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:32.951964    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/418b9f85-2272-4477-ae6b-196bb94fb85b-kube-api-access-dllcs" (OuterVolumeSpecName: "kube-api-access-dllcs") pod "418b9f85-2272-4477-ae6b-196bb94fb85b" (UID: "418b9f85-2272-4477-ae6b-196bb94fb85b"). InnerVolumeSpecName "kube-api-access-dllcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 04:02:33.051333 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:33.051306    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dllcs\" (UniqueName: \"kubernetes.io/projected/418b9f85-2272-4477-ae6b-196bb94fb85b-kube-api-access-dllcs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:02:33.051333 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:33.051330    2570 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/418b9f85-2272-4477-ae6b-196bb94fb85b-extensions-socket-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:02:33.077303 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:33.077282    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p5www"]
Jul 29 04:02:33.081609 ip-10-0-135-138 kubenswrapper[2570]: W0729 04:02:33.081579    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode325693f_db11_4d81_8ddc_3b3d4c5c35a2.slice/crio-4d920d05faf0152bce0ccb6f4f31fc464eb23991b53fafa58e06cea99f385e12 WatchSource:0}: Error finding container 4d920d05faf0152bce0ccb6f4f31fc464eb23991b53fafa58e06cea99f385e12: Status 404 returned error can't find the container with id 4d920d05faf0152bce0ccb6f4f31fc464eb23991b53fafa58e06cea99f385e12
Jul 29 04:02:33.210484 ip-10-0-135-138 kubenswrapper[2570]: E0729 04:02:33.210456    2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6x9lh\" is forbidden: User \"system:node:ip-10-0-135-138.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-138.ec2.internal' and this object" podUID="418b9f85-2272-4477-ae6b-196bb94fb85b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6x9lh"
Jul 29 04:02:33.834890 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:33.834858    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="418b9f85-2272-4477-ae6b-196bb94fb85b" path="/var/lib/kubelet/pods/418b9f85-2272-4477-ae6b-196bb94fb85b/volumes"
Jul 29 04:02:33.905489 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:33.905455    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p5www" event={"ID":"e325693f-db11-4d81-8ddc-3b3d4c5c35a2","Type":"ContainerStarted","Data":"995c7a1f3ef30c0c8951c292b235becab4965c5b688b04c886d8577ebdf6834e"}
Jul 29 04:02:33.905646 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:33.905502    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p5www" event={"ID":"e325693f-db11-4d81-8ddc-3b3d4c5c35a2","Type":"ContainerStarted","Data":"4d920d05faf0152bce0ccb6f4f31fc464eb23991b53fafa58e06cea99f385e12"}
Jul 29 04:02:33.905646 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:33.905568    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p5www"
Jul 29 04:02:33.931297 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:33.931251    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p5www" podStartSLOduration=1.9312377779999998 podStartE2EDuration="1.931237778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:02:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:02:33.928990188 +0000 UTC m=+418.659121257" watchObservedRunningTime="2026-07-29 04:02:33.931237778 +0000 UTC m=+418.661368848"
Jul 29 04:02:44.913544 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:44.913516    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p5www"
Jul 29 04:02:48.593085 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:48.593051    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p5www"]
Jul 29 04:02:48.593523 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:48.593279    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p5www" podUID="e325693f-db11-4d81-8ddc-3b3d4c5c35a2" containerName="manager" containerID="cri-o://995c7a1f3ef30c0c8951c292b235becab4965c5b688b04c886d8577ebdf6834e" gracePeriod=10
Jul 29 04:02:48.813555 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:48.813521    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rxc52"]
Jul 29 04:02:48.815934 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:48.815918    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rxc52"
Jul 29 04:02:48.834425 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:48.834397    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rxc52"]
Jul 29 04:02:48.889675 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:48.889603    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq6mk\" (UniqueName: \"kubernetes.io/projected/6a0da2f7-c76c-4502-9977-be5904616395-kube-api-access-sq6mk\") pod \"kuadrant-operator-controller-manager-86878c769f-rxc52\" (UID: \"6a0da2f7-c76c-4502-9977-be5904616395\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rxc52"
Jul 29 04:02:48.889675 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:48.889659    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6a0da2f7-c76c-4502-9977-be5904616395-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-rxc52\" (UID: \"6a0da2f7-c76c-4502-9977-be5904616395\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rxc52"
Jul 29 04:02:48.973560 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:48.973530    2570 generic.go:356] "Generic (PLEG): container finished" podID="e325693f-db11-4d81-8ddc-3b3d4c5c35a2" containerID="995c7a1f3ef30c0c8951c292b235becab4965c5b688b04c886d8577ebdf6834e" exitCode=0
Jul 29 04:02:48.973718 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:48.973586    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p5www" event={"ID":"e325693f-db11-4d81-8ddc-3b3d4c5c35a2","Type":"ContainerDied","Data":"995c7a1f3ef30c0c8951c292b235becab4965c5b688b04c886d8577ebdf6834e"}
Jul 29 04:02:48.990497 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:48.990470    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq6mk\" (UniqueName: \"kubernetes.io/projected/6a0da2f7-c76c-4502-9977-be5904616395-kube-api-access-sq6mk\") pod \"kuadrant-operator-controller-manager-86878c769f-rxc52\" (UID: \"6a0da2f7-c76c-4502-9977-be5904616395\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rxc52"
Jul 29 04:02:48.990593 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:48.990526    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6a0da2f7-c76c-4502-9977-be5904616395-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-rxc52\" (UID: \"6a0da2f7-c76c-4502-9977-be5904616395\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rxc52"
Jul 29 04:02:48.990842 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:48.990806    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6a0da2f7-c76c-4502-9977-be5904616395-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-rxc52\" (UID: \"6a0da2f7-c76c-4502-9977-be5904616395\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rxc52"
Jul 29 04:02:49.002566 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:49.002540    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq6mk\" (UniqueName: \"kubernetes.io/projected/6a0da2f7-c76c-4502-9977-be5904616395-kube-api-access-sq6mk\") pod \"kuadrant-operator-controller-manager-86878c769f-rxc52\" (UID: \"6a0da2f7-c76c-4502-9977-be5904616395\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rxc52"
Jul 29 04:02:49.125957 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:49.125927    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rxc52"
Jul 29 04:02:49.256703 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:49.256678    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rxc52"]
Jul 29 04:02:49.259281 ip-10-0-135-138 kubenswrapper[2570]: W0729 04:02:49.259256    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a0da2f7_c76c_4502_9977_be5904616395.slice/crio-4cf3ea1fdcac21fae2c9e2ecdf2febeb5558414d60b0334aeeef44daa6947cc1 WatchSource:0}: Error finding container 4cf3ea1fdcac21fae2c9e2ecdf2febeb5558414d60b0334aeeef44daa6947cc1: Status 404 returned error can't find the container with id 4cf3ea1fdcac21fae2c9e2ecdf2febeb5558414d60b0334aeeef44daa6947cc1
Jul 29 04:02:49.330139 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:49.330115    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p5www"
Jul 29 04:02:49.394676 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:49.394643    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d9hnn\" (UniqueName: \"kubernetes.io/projected/e325693f-db11-4d81-8ddc-3b3d4c5c35a2-kube-api-access-d9hnn\") pod \"e325693f-db11-4d81-8ddc-3b3d4c5c35a2\" (UID: \"e325693f-db11-4d81-8ddc-3b3d4c5c35a2\") "
Jul 29 04:02:49.394848 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:49.394768    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e325693f-db11-4d81-8ddc-3b3d4c5c35a2-extensions-socket-volume\") pod \"e325693f-db11-4d81-8ddc-3b3d4c5c35a2\" (UID: \"e325693f-db11-4d81-8ddc-3b3d4c5c35a2\") "
Jul 29 04:02:49.395088 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:49.395065    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e325693f-db11-4d81-8ddc-3b3d4c5c35a2-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "e325693f-db11-4d81-8ddc-3b3d4c5c35a2" (UID: "e325693f-db11-4d81-8ddc-3b3d4c5c35a2"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 04:02:49.396861 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:49.396840    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e325693f-db11-4d81-8ddc-3b3d4c5c35a2-kube-api-access-d9hnn" (OuterVolumeSpecName: "kube-api-access-d9hnn") pod "e325693f-db11-4d81-8ddc-3b3d4c5c35a2" (UID: "e325693f-db11-4d81-8ddc-3b3d4c5c35a2"). InnerVolumeSpecName "kube-api-access-d9hnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 04:02:49.495604 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:49.495536    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9hnn\" (UniqueName: \"kubernetes.io/projected/e325693f-db11-4d81-8ddc-3b3d4c5c35a2-kube-api-access-d9hnn\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:02:49.495604 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:49.495563    2570 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e325693f-db11-4d81-8ddc-3b3d4c5c35a2-extensions-socket-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:02:49.978917 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:49.978885    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p5www"
Jul 29 04:02:49.979363 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:49.978915    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p5www" event={"ID":"e325693f-db11-4d81-8ddc-3b3d4c5c35a2","Type":"ContainerDied","Data":"4d920d05faf0152bce0ccb6f4f31fc464eb23991b53fafa58e06cea99f385e12"}
Jul 29 04:02:49.979363 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:49.978975    2570 scope.go:117] "RemoveContainer" containerID="995c7a1f3ef30c0c8951c292b235becab4965c5b688b04c886d8577ebdf6834e"
Jul 29 04:02:49.980597 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:49.980573    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rxc52" event={"ID":"6a0da2f7-c76c-4502-9977-be5904616395","Type":"ContainerStarted","Data":"ae4304475f5f9dfd9d039d0a322ab44702256f3c973b7fe3ff760784aff37164"}
Jul 29 04:02:49.980721 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:49.980621    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rxc52" event={"ID":"6a0da2f7-c76c-4502-9977-be5904616395","Type":"ContainerStarted","Data":"4cf3ea1fdcac21fae2c9e2ecdf2febeb5558414d60b0334aeeef44daa6947cc1"}
Jul 29 04:02:49.980802 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:49.980786    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rxc52"
Jul 29 04:02:50.001182 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:50.001159    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p5www"]
Jul 29 04:02:50.008635 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:50.008614    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p5www"]
Jul 29 04:02:50.032001 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:50.031963    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rxc52" podStartSLOduration=2.031949883 podStartE2EDuration="2.031949883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:02:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:02:50.030884887 +0000 UTC m=+434.761015975" watchObservedRunningTime="2026-07-29 04:02:50.031949883 +0000 UTC m=+434.762080953"
Jul 29 04:02:51.834206 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:02:51.834175    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e325693f-db11-4d81-8ddc-3b3d4c5c35a2" path="/var/lib/kubelet/pods/e325693f-db11-4d81-8ddc-3b3d4c5c35a2/volumes"
Jul 29 04:03:00.988335 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:00.988259    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rxc52"
Jul 29 04:03:09.239708 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:09.239675    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-9vwpb"]
Jul 29 04:03:09.240130 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:09.240049    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e325693f-db11-4d81-8ddc-3b3d4c5c35a2" containerName="manager"
Jul 29 04:03:09.240130 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:09.240060    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="e325693f-db11-4d81-8ddc-3b3d4c5c35a2" containerName="manager"
Jul 29 04:03:09.240130 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:09.240117    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="e325693f-db11-4d81-8ddc-3b3d4c5c35a2" containerName="manager"
Jul 29 04:03:09.242180 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:09.242160    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-9vwpb"
Jul 29 04:03:09.245176 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:09.245157    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-dht7f\""
Jul 29 04:03:09.251783 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:09.251761    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-9vwpb"]
Jul 29 04:03:09.372983 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:09.372954    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j8fv\" (UniqueName: \"kubernetes.io/projected/f984d27c-9c8b-4cad-8d07-47b7218ea6f2-kube-api-access-8j8fv\") pod \"authorino-58cd955675-9vwpb\" (UID: \"f984d27c-9c8b-4cad-8d07-47b7218ea6f2\") " pod="kuadrant-system/authorino-58cd955675-9vwpb"
Jul 29 04:03:09.443598 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:09.443562    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-prlxq"]
Jul 29 04:03:09.446201 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:09.446184    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-prlxq"
Jul 29 04:03:09.453585 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:09.453565    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-prlxq"]
Jul 29 04:03:09.474115 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:09.474087    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j8fv\" (UniqueName: \"kubernetes.io/projected/f984d27c-9c8b-4cad-8d07-47b7218ea6f2-kube-api-access-8j8fv\") pod \"authorino-58cd955675-9vwpb\" (UID: \"f984d27c-9c8b-4cad-8d07-47b7218ea6f2\") " pod="kuadrant-system/authorino-58cd955675-9vwpb"
Jul 29 04:03:09.482478 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:09.482455    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j8fv\" (UniqueName: \"kubernetes.io/projected/f984d27c-9c8b-4cad-8d07-47b7218ea6f2-kube-api-access-8j8fv\") pod \"authorino-58cd955675-9vwpb\" (UID: \"f984d27c-9c8b-4cad-8d07-47b7218ea6f2\") " pod="kuadrant-system/authorino-58cd955675-9vwpb"
Jul 29 04:03:09.553534 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:09.553478    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-9vwpb"
Jul 29 04:03:09.574625 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:09.574601    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9z9d\" (UniqueName: \"kubernetes.io/projected/59fb94fa-8d20-42b7-8f94-136ff4a8d778-kube-api-access-g9z9d\") pod \"authorino-75d556bd77-prlxq\" (UID: \"59fb94fa-8d20-42b7-8f94-136ff4a8d778\") " pod="kuadrant-system/authorino-75d556bd77-prlxq"
Jul 29 04:03:09.675437 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:09.675402    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9z9d\" (UniqueName: \"kubernetes.io/projected/59fb94fa-8d20-42b7-8f94-136ff4a8d778-kube-api-access-g9z9d\") pod \"authorino-75d556bd77-prlxq\" (UID: \"59fb94fa-8d20-42b7-8f94-136ff4a8d778\") " pod="kuadrant-system/authorino-75d556bd77-prlxq"
Jul 29 04:03:09.678227 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:09.678204    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-9vwpb"]
Jul 29 04:03:09.680597 ip-10-0-135-138 kubenswrapper[2570]: W0729 04:03:09.680574    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf984d27c_9c8b_4cad_8d07_47b7218ea6f2.slice/crio-4eca089cc0dbd6faa150fbbc41e5c5158b3ee2217a17787d5db89fc6d4fc87eb WatchSource:0}: Error finding container 4eca089cc0dbd6faa150fbbc41e5c5158b3ee2217a17787d5db89fc6d4fc87eb: Status 404 returned error can't find the container with id 4eca089cc0dbd6faa150fbbc41e5c5158b3ee2217a17787d5db89fc6d4fc87eb
Jul 29 04:03:09.683920 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:09.683898    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9z9d\" (UniqueName: \"kubernetes.io/projected/59fb94fa-8d20-42b7-8f94-136ff4a8d778-kube-api-access-g9z9d\") pod \"authorino-75d556bd77-prlxq\" (UID: \"59fb94fa-8d20-42b7-8f94-136ff4a8d778\") " pod="kuadrant-system/authorino-75d556bd77-prlxq"
Jul 29 04:03:09.755675 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:09.755644    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-prlxq"
Jul 29 04:03:09.883270 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:09.883244    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-prlxq"]
Jul 29 04:03:09.885145 ip-10-0-135-138 kubenswrapper[2570]: W0729 04:03:09.885122    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod59fb94fa_8d20_42b7_8f94_136ff4a8d778.slice/crio-5151d02d36d24d50411ed7624638a47fc60f959cc57c0c4d049083995373080b WatchSource:0}: Error finding container 5151d02d36d24d50411ed7624638a47fc60f959cc57c0c4d049083995373080b: Status 404 returned error can't find the container with id 5151d02d36d24d50411ed7624638a47fc60f959cc57c0c4d049083995373080b
Jul 29 04:03:10.073478 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:10.073442    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-9vwpb" event={"ID":"f984d27c-9c8b-4cad-8d07-47b7218ea6f2","Type":"ContainerStarted","Data":"4eca089cc0dbd6faa150fbbc41e5c5158b3ee2217a17787d5db89fc6d4fc87eb"}
Jul 29 04:03:10.074556 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:10.074529    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-prlxq" event={"ID":"59fb94fa-8d20-42b7-8f94-136ff4a8d778","Type":"ContainerStarted","Data":"5151d02d36d24d50411ed7624638a47fc60f959cc57c0c4d049083995373080b"}
Jul 29 04:03:14.098582 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:14.098532    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-9vwpb" event={"ID":"f984d27c-9c8b-4cad-8d07-47b7218ea6f2","Type":"ContainerStarted","Data":"90f3f9f1f27aa787a76b6d41955ea15860e17a9fd9f3700e2b7c783070efd7d0"}
Jul 29 04:03:14.100068 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:14.100042    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-prlxq" event={"ID":"59fb94fa-8d20-42b7-8f94-136ff4a8d778","Type":"ContainerStarted","Data":"724193a508405d248a94a53820f3d85c7d44bc8822abf38e0723af8761c7ef31"}
Jul 29 04:03:14.114546 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:14.114502    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-9vwpb" podStartSLOduration=1.666756763 podStartE2EDuration="5.114489481s" totalImagesPullingTime="3.447732718s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:03:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:03:14.11276266 +0000 UTC m=+458.842893734" watchObservedRunningTime="2026-07-29 04:03:14.114489481 +0000 UTC m=+458.844620550"
Jul 29 04:03:14.149677 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:14.149615    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-prlxq" podStartSLOduration=1.89492483 podStartE2EDuration="5.149594935s" totalImagesPullingTime="3.254670105s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:03:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:03:14.126235034 +0000 UTC m=+458.856366104" watchObservedRunningTime="2026-07-29 04:03:14.149594935 +0000 UTC m=+458.879726006"
Jul 29 04:03:14.157840 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:14.151570    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-9vwpb"]
Jul 29 04:03:16.109449 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:16.109381    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-9vwpb" podUID="f984d27c-9c8b-4cad-8d07-47b7218ea6f2" containerName="authorino" containerID="cri-o://90f3f9f1f27aa787a76b6d41955ea15860e17a9fd9f3700e2b7c783070efd7d0" gracePeriod=30
Jul 29 04:03:16.355406 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:16.355382    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-9vwpb"
Jul 29 04:03:16.434854 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:16.434747    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8j8fv\" (UniqueName: \"kubernetes.io/projected/f984d27c-9c8b-4cad-8d07-47b7218ea6f2-kube-api-access-8j8fv\") pod \"f984d27c-9c8b-4cad-8d07-47b7218ea6f2\" (UID: \"f984d27c-9c8b-4cad-8d07-47b7218ea6f2\") "
Jul 29 04:03:16.436977 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:16.436943    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f984d27c-9c8b-4cad-8d07-47b7218ea6f2-kube-api-access-8j8fv" (OuterVolumeSpecName: "kube-api-access-8j8fv") pod "f984d27c-9c8b-4cad-8d07-47b7218ea6f2" (UID: "f984d27c-9c8b-4cad-8d07-47b7218ea6f2"). InnerVolumeSpecName "kube-api-access-8j8fv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 04:03:16.535749 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:16.535711    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8j8fv\" (UniqueName: \"kubernetes.io/projected/f984d27c-9c8b-4cad-8d07-47b7218ea6f2-kube-api-access-8j8fv\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:03:17.114887 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:17.114851    2570 generic.go:356] "Generic (PLEG): container finished" podID="f984d27c-9c8b-4cad-8d07-47b7218ea6f2" containerID="90f3f9f1f27aa787a76b6d41955ea15860e17a9fd9f3700e2b7c783070efd7d0" exitCode=0
Jul 29 04:03:17.114887 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:17.114880    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-9vwpb"
Jul 29 04:03:17.115408 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:17.114919    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-9vwpb" event={"ID":"f984d27c-9c8b-4cad-8d07-47b7218ea6f2","Type":"ContainerDied","Data":"90f3f9f1f27aa787a76b6d41955ea15860e17a9fd9f3700e2b7c783070efd7d0"}
Jul 29 04:03:17.115408 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:17.114957    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-9vwpb" event={"ID":"f984d27c-9c8b-4cad-8d07-47b7218ea6f2","Type":"ContainerDied","Data":"4eca089cc0dbd6faa150fbbc41e5c5158b3ee2217a17787d5db89fc6d4fc87eb"}
Jul 29 04:03:17.115408 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:17.114974    2570 scope.go:117] "RemoveContainer" containerID="90f3f9f1f27aa787a76b6d41955ea15860e17a9fd9f3700e2b7c783070efd7d0"
Jul 29 04:03:17.124199 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:17.124185    2570 scope.go:117] "RemoveContainer" containerID="90f3f9f1f27aa787a76b6d41955ea15860e17a9fd9f3700e2b7c783070efd7d0"
Jul 29 04:03:17.124462 ip-10-0-135-138 kubenswrapper[2570]: E0729 04:03:17.124439    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90f3f9f1f27aa787a76b6d41955ea15860e17a9fd9f3700e2b7c783070efd7d0\": container with ID starting with 90f3f9f1f27aa787a76b6d41955ea15860e17a9fd9f3700e2b7c783070efd7d0 not found: ID does not exist" containerID="90f3f9f1f27aa787a76b6d41955ea15860e17a9fd9f3700e2b7c783070efd7d0"
Jul 29 04:03:17.124546 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:17.124466    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90f3f9f1f27aa787a76b6d41955ea15860e17a9fd9f3700e2b7c783070efd7d0"} err="failed to get container status \"90f3f9f1f27aa787a76b6d41955ea15860e17a9fd9f3700e2b7c783070efd7d0\": rpc error: code = NotFound desc = could not find container \"90f3f9f1f27aa787a76b6d41955ea15860e17a9fd9f3700e2b7c783070efd7d0\": container with ID starting with 90f3f9f1f27aa787a76b6d41955ea15860e17a9fd9f3700e2b7c783070efd7d0 not found: ID does not exist"
Jul 29 04:03:17.136208 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:17.136185    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-9vwpb"]
Jul 29 04:03:17.139740 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:17.139719    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-9vwpb"]
Jul 29 04:03:17.835169 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:17.835136    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f984d27c-9c8b-4cad-8d07-47b7218ea6f2" path="/var/lib/kubelet/pods/f984d27c-9c8b-4cad-8d07-47b7218ea6f2/volumes"
Jul 29 04:03:27.642526 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:27.642487    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-8677fddb6d-qrwsn"]
Jul 29 04:03:27.642986 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:27.642901    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f984d27c-9c8b-4cad-8d07-47b7218ea6f2" containerName="authorino"
Jul 29 04:03:27.642986 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:27.642915    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="f984d27c-9c8b-4cad-8d07-47b7218ea6f2" containerName="authorino"
Jul 29 04:03:27.642986 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:27.642986    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="f984d27c-9c8b-4cad-8d07-47b7218ea6f2" containerName="authorino"
Jul 29 04:03:27.645964 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:27.645944    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8677fddb6d-qrwsn"
Jul 29 04:03:27.657696 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:27.657675    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8677fddb6d-qrwsn"]
Jul 29 04:03:27.837163 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:27.837135    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c5695ea5-b8ee-4dbd-a7d4-e1a97ef0cf8b-console-serving-cert\") pod \"console-8677fddb6d-qrwsn\" (UID: \"c5695ea5-b8ee-4dbd-a7d4-e1a97ef0cf8b\") " pod="openshift-console/console-8677fddb6d-qrwsn"
Jul 29 04:03:27.837315 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:27.837180    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c5695ea5-b8ee-4dbd-a7d4-e1a97ef0cf8b-service-ca\") pod \"console-8677fddb6d-qrwsn\" (UID: \"c5695ea5-b8ee-4dbd-a7d4-e1a97ef0cf8b\") " pod="openshift-console/console-8677fddb6d-qrwsn"
Jul 29 04:03:27.837315 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:27.837197    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5695ea5-b8ee-4dbd-a7d4-e1a97ef0cf8b-trusted-ca-bundle\") pod \"console-8677fddb6d-qrwsn\" (UID: \"c5695ea5-b8ee-4dbd-a7d4-e1a97ef0cf8b\") " pod="openshift-console/console-8677fddb6d-qrwsn"
Jul 29 04:03:27.837315 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:27.837234    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqh8f\" (UniqueName: \"kubernetes.io/projected/c5695ea5-b8ee-4dbd-a7d4-e1a97ef0cf8b-kube-api-access-pqh8f\") pod \"console-8677fddb6d-qrwsn\" (UID: \"c5695ea5-b8ee-4dbd-a7d4-e1a97ef0cf8b\") " pod="openshift-console/console-8677fddb6d-qrwsn"
Jul 29 04:03:27.837315 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:27.837292    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c5695ea5-b8ee-4dbd-a7d4-e1a97ef0cf8b-console-oauth-config\") pod \"console-8677fddb6d-qrwsn\" (UID: \"c5695ea5-b8ee-4dbd-a7d4-e1a97ef0cf8b\") " pod="openshift-console/console-8677fddb6d-qrwsn"
Jul 29 04:03:27.837315 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:27.837309    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c5695ea5-b8ee-4dbd-a7d4-e1a97ef0cf8b-console-config\") pod \"console-8677fddb6d-qrwsn\" (UID: \"c5695ea5-b8ee-4dbd-a7d4-e1a97ef0cf8b\") " pod="openshift-console/console-8677fddb6d-qrwsn"
Jul 29 04:03:27.837475 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:27.837400    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c5695ea5-b8ee-4dbd-a7d4-e1a97ef0cf8b-oauth-serving-cert\") pod \"console-8677fddb6d-qrwsn\" (UID: \"c5695ea5-b8ee-4dbd-a7d4-e1a97ef0cf8b\") " pod="openshift-console/console-8677fddb6d-qrwsn"
Jul 29 04:03:27.937857 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:27.937754    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c5695ea5-b8ee-4dbd-a7d4-e1a97ef0cf8b-service-ca\") pod \"console-8677fddb6d-qrwsn\" (UID: \"c5695ea5-b8ee-4dbd-a7d4-e1a97ef0cf8b\") " pod="openshift-console/console-8677fddb6d-qrwsn"
Jul 29 04:03:27.937857 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:27.937784    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5695ea5-b8ee-4dbd-a7d4-e1a97ef0cf8b-trusted-ca-bundle\") pod \"console-8677fddb6d-qrwsn\" (UID: \"c5695ea5-b8ee-4dbd-a7d4-e1a97ef0cf8b\") " pod="openshift-console/console-8677fddb6d-qrwsn"
Jul 29 04:03:27.937857 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:27.937802    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqh8f\" (UniqueName: \"kubernetes.io/projected/c5695ea5-b8ee-4dbd-a7d4-e1a97ef0cf8b-kube-api-access-pqh8f\") pod \"console-8677fddb6d-qrwsn\" (UID: \"c5695ea5-b8ee-4dbd-a7d4-e1a97ef0cf8b\") " pod="openshift-console/console-8677fddb6d-qrwsn"
Jul 29 04:03:27.938094 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:27.937966    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c5695ea5-b8ee-4dbd-a7d4-e1a97ef0cf8b-console-oauth-config\") pod \"console-8677fddb6d-qrwsn\" (UID: \"c5695ea5-b8ee-4dbd-a7d4-e1a97ef0cf8b\") " pod="openshift-console/console-8677fddb6d-qrwsn"
Jul 29 04:03:27.938094 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:27.937989    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c5695ea5-b8ee-4dbd-a7d4-e1a97ef0cf8b-console-config\") pod \"console-8677fddb6d-qrwsn\" (UID: \"c5695ea5-b8ee-4dbd-a7d4-e1a97ef0cf8b\") " pod="openshift-console/console-8677fddb6d-qrwsn"
Jul 29 04:03:27.938182 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:27.938110    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c5695ea5-b8ee-4dbd-a7d4-e1a97ef0cf8b-oauth-serving-cert\") pod \"console-8677fddb6d-qrwsn\" (UID: \"c5695ea5-b8ee-4dbd-a7d4-e1a97ef0cf8b\") " pod="openshift-console/console-8677fddb6d-qrwsn"
Jul 29 04:03:27.938182 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:27.938152    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c5695ea5-b8ee-4dbd-a7d4-e1a97ef0cf8b-console-serving-cert\") pod \"console-8677fddb6d-qrwsn\" (UID: \"c5695ea5-b8ee-4dbd-a7d4-e1a97ef0cf8b\") " pod="openshift-console/console-8677fddb6d-qrwsn"
Jul 29 04:03:27.938649 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:27.938624    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c5695ea5-b8ee-4dbd-a7d4-e1a97ef0cf8b-service-ca\") pod \"console-8677fddb6d-qrwsn\" (UID: \"c5695ea5-b8ee-4dbd-a7d4-e1a97ef0cf8b\") " pod="openshift-console/console-8677fddb6d-qrwsn"
Jul 29 04:03:27.938797 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:27.938685    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c5695ea5-b8ee-4dbd-a7d4-e1a97ef0cf8b-console-config\") pod \"console-8677fddb6d-qrwsn\" (UID: \"c5695ea5-b8ee-4dbd-a7d4-e1a97ef0cf8b\") " pod="openshift-console/console-8677fddb6d-qrwsn"
Jul 29 04:03:27.938893 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:27.938730    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c5695ea5-b8ee-4dbd-a7d4-e1a97ef0cf8b-oauth-serving-cert\") pod \"console-8677fddb6d-qrwsn\" (UID: \"c5695ea5-b8ee-4dbd-a7d4-e1a97ef0cf8b\") " pod="openshift-console/console-8677fddb6d-qrwsn"
Jul 29 04:03:27.938982 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:27.938965    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5695ea5-b8ee-4dbd-a7d4-e1a97ef0cf8b-trusted-ca-bundle\") pod \"console-8677fddb6d-qrwsn\" (UID: \"c5695ea5-b8ee-4dbd-a7d4-e1a97ef0cf8b\") " pod="openshift-console/console-8677fddb6d-qrwsn"
Jul 29 04:03:27.940596 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:27.940578    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c5695ea5-b8ee-4dbd-a7d4-e1a97ef0cf8b-console-serving-cert\") pod \"console-8677fddb6d-qrwsn\" (UID: \"c5695ea5-b8ee-4dbd-a7d4-e1a97ef0cf8b\") " pod="openshift-console/console-8677fddb6d-qrwsn"
Jul 29 04:03:27.940654 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:27.940629    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c5695ea5-b8ee-4dbd-a7d4-e1a97ef0cf8b-console-oauth-config\") pod \"console-8677fddb6d-qrwsn\" (UID: \"c5695ea5-b8ee-4dbd-a7d4-e1a97ef0cf8b\") " pod="openshift-console/console-8677fddb6d-qrwsn"
Jul 29 04:03:27.949596 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:27.949576    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqh8f\" (UniqueName: \"kubernetes.io/projected/c5695ea5-b8ee-4dbd-a7d4-e1a97ef0cf8b-kube-api-access-pqh8f\") pod \"console-8677fddb6d-qrwsn\" (UID: \"c5695ea5-b8ee-4dbd-a7d4-e1a97ef0cf8b\") " pod="openshift-console/console-8677fddb6d-qrwsn"
Jul 29 04:03:27.955470 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:27.955451    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8677fddb6d-qrwsn"
Jul 29 04:03:28.106248 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:28.106225    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8677fddb6d-qrwsn"]
Jul 29 04:03:28.108849 ip-10-0-135-138 kubenswrapper[2570]: W0729 04:03:28.108802    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5695ea5_b8ee_4dbd_a7d4_e1a97ef0cf8b.slice/crio-ae24a683d5eab060a87c22c2a114e58e110cc04649105b32305a1dcdc402ca25 WatchSource:0}: Error finding container ae24a683d5eab060a87c22c2a114e58e110cc04649105b32305a1dcdc402ca25: Status 404 returned error can't find the container with id ae24a683d5eab060a87c22c2a114e58e110cc04649105b32305a1dcdc402ca25
Jul 29 04:03:28.177232 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:28.177201    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8677fddb6d-qrwsn" event={"ID":"c5695ea5-b8ee-4dbd-a7d4-e1a97ef0cf8b","Type":"ContainerStarted","Data":"ae24a683d5eab060a87c22c2a114e58e110cc04649105b32305a1dcdc402ca25"}
Jul 29 04:03:29.184056 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:29.184020    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8677fddb6d-qrwsn" event={"ID":"c5695ea5-b8ee-4dbd-a7d4-e1a97ef0cf8b","Type":"ContainerStarted","Data":"7fc97d0c971c79038b2a2f7cdce4ac452f3c053b190a09befca58b1236101af6"}
Jul 29 04:03:29.204991 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:29.204940    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-8677fddb6d-qrwsn" podStartSLOduration=2.204924834 podStartE2EDuration="2.204924834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:03:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:03:29.203221115 +0000 UTC m=+473.933352212" watchObservedRunningTime="2026-07-29 04:03:29.204924834 +0000 UTC m=+473.935055903"
Jul 29 04:03:37.956580 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:37.956550    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8677fddb6d-qrwsn"
Jul 29 04:03:37.956991 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:37.956603    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8677fddb6d-qrwsn"
Jul 29 04:03:37.961193 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:37.961174    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8677fddb6d-qrwsn"
Jul 29 04:03:38.230447 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:38.230379    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8677fddb6d-qrwsn"
Jul 29 04:03:38.277873 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:38.277845    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-598469cf5b-ttwbp"]
Jul 29 04:03:41.456502 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:41.456465    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-t9njz"]
Jul 29 04:03:41.461489 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:41.461458    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-t9njz"
Jul 29 04:03:41.471131 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:41.471105    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-t9njz"]
Jul 29 04:03:41.551917 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:41.551879    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8dps\" (UniqueName: \"kubernetes.io/projected/6eb29ed3-8591-4dd8-b582-1f8185645af5-kube-api-access-l8dps\") pod \"authorino-5d4b67c446-t9njz\" (UID: \"6eb29ed3-8591-4dd8-b582-1f8185645af5\") " pod="kuadrant-system/authorino-5d4b67c446-t9njz"
Jul 29 04:03:41.628180 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:41.628144    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-t9njz"]
Jul 29 04:03:41.628462 ip-10-0-135-138 kubenswrapper[2570]: E0729 04:03:41.628440    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-l8dps], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-t9njz" podUID="6eb29ed3-8591-4dd8-b582-1f8185645af5"
Jul 29 04:03:41.652789 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:41.652752    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8dps\" (UniqueName: \"kubernetes.io/projected/6eb29ed3-8591-4dd8-b582-1f8185645af5-kube-api-access-l8dps\") pod \"authorino-5d4b67c446-t9njz\" (UID: \"6eb29ed3-8591-4dd8-b582-1f8185645af5\") " pod="kuadrant-system/authorino-5d4b67c446-t9njz"
Jul 29 04:03:41.668104 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:41.668073    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8dps\" (UniqueName: \"kubernetes.io/projected/6eb29ed3-8591-4dd8-b582-1f8185645af5-kube-api-access-l8dps\") pod \"authorino-5d4b67c446-t9njz\" (UID: \"6eb29ed3-8591-4dd8-b582-1f8185645af5\") " pod="kuadrant-system/authorino-5d4b67c446-t9njz"
Jul 29 04:03:42.243737 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:42.243706    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-t9njz"
Jul 29 04:03:42.250014 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:42.249990    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-t9njz"
Jul 29 04:03:42.358543 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:42.358511    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l8dps\" (UniqueName: \"kubernetes.io/projected/6eb29ed3-8591-4dd8-b582-1f8185645af5-kube-api-access-l8dps\") pod \"6eb29ed3-8591-4dd8-b582-1f8185645af5\" (UID: \"6eb29ed3-8591-4dd8-b582-1f8185645af5\") "
Jul 29 04:03:42.360721 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:42.360698    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6eb29ed3-8591-4dd8-b582-1f8185645af5-kube-api-access-l8dps" (OuterVolumeSpecName: "kube-api-access-l8dps") pod "6eb29ed3-8591-4dd8-b582-1f8185645af5" (UID: "6eb29ed3-8591-4dd8-b582-1f8185645af5"). InnerVolumeSpecName "kube-api-access-l8dps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 04:03:42.462289 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:42.462254    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8dps\" (UniqueName: \"kubernetes.io/projected/6eb29ed3-8591-4dd8-b582-1f8185645af5-kube-api-access-l8dps\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:03:43.248601 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:43.248570    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-t9njz"
Jul 29 04:03:43.286512 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:43.286479    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-t9njz"]
Jul 29 04:03:43.293486 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:43.293461    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-t9njz"]
Jul 29 04:03:43.834619 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:43.834587    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6eb29ed3-8591-4dd8-b582-1f8185645af5" path="/var/lib/kubelet/pods/6eb29ed3-8591-4dd8-b582-1f8185645af5/volumes"
Jul 29 04:03:46.041930 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:46.041898    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-prlxq"]
Jul 29 04:03:46.042391 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:46.042125    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-prlxq" podUID="59fb94fa-8d20-42b7-8f94-136ff4a8d778" containerName="authorino" containerID="cri-o://724193a508405d248a94a53820f3d85c7d44bc8822abf38e0723af8761c7ef31" gracePeriod=30
Jul 29 04:03:46.265343 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:46.265307    2570 generic.go:356] "Generic (PLEG): container finished" podID="59fb94fa-8d20-42b7-8f94-136ff4a8d778" containerID="724193a508405d248a94a53820f3d85c7d44bc8822abf38e0723af8761c7ef31" exitCode=0
Jul 29 04:03:46.265494 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:46.265377    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-prlxq" event={"ID":"59fb94fa-8d20-42b7-8f94-136ff4a8d778","Type":"ContainerDied","Data":"724193a508405d248a94a53820f3d85c7d44bc8822abf38e0723af8761c7ef31"}
Jul 29 04:03:46.287508 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:46.287480    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-prlxq"
Jul 29 04:03:46.396416 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:46.396386    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g9z9d\" (UniqueName: \"kubernetes.io/projected/59fb94fa-8d20-42b7-8f94-136ff4a8d778-kube-api-access-g9z9d\") pod \"59fb94fa-8d20-42b7-8f94-136ff4a8d778\" (UID: \"59fb94fa-8d20-42b7-8f94-136ff4a8d778\") "
Jul 29 04:03:46.398550 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:46.398525    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59fb94fa-8d20-42b7-8f94-136ff4a8d778-kube-api-access-g9z9d" (OuterVolumeSpecName: "kube-api-access-g9z9d") pod "59fb94fa-8d20-42b7-8f94-136ff4a8d778" (UID: "59fb94fa-8d20-42b7-8f94-136ff4a8d778"). InnerVolumeSpecName "kube-api-access-g9z9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 04:03:46.497699 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:46.497667    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9z9d\" (UniqueName: \"kubernetes.io/projected/59fb94fa-8d20-42b7-8f94-136ff4a8d778-kube-api-access-g9z9d\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:03:47.271882 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:47.271844    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-prlxq"
Jul 29 04:03:47.271882 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:47.271866    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-prlxq" event={"ID":"59fb94fa-8d20-42b7-8f94-136ff4a8d778","Type":"ContainerDied","Data":"5151d02d36d24d50411ed7624638a47fc60f959cc57c0c4d049083995373080b"}
Jul 29 04:03:47.272483 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:47.271922    2570 scope.go:117] "RemoveContainer" containerID="724193a508405d248a94a53820f3d85c7d44bc8822abf38e0723af8761c7ef31"
Jul 29 04:03:47.307038 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:47.307009    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-prlxq"]
Jul 29 04:03:47.315845 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:47.309329    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-prlxq"]
Jul 29 04:03:47.834239 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:03:47.834207    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59fb94fa-8d20-42b7-8f94-136ff4a8d778" path="/var/lib/kubelet/pods/59fb94fa-8d20-42b7-8f94-136ff4a8d778/volumes"
Jul 29 04:04:03.297800 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:04:03.297734    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-598469cf5b-ttwbp" podUID="072b4622-e276-45df-80ae-b870afbbea26" containerName="console" containerID="cri-o://f810ba5b7830329a9fb221ddbd547aa1f8d415638cfa57a3522bc4a33d43b3ec" gracePeriod=15
Jul 29 04:04:03.538561 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:04:03.538536    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-598469cf5b-ttwbp"
Jul 29 04:04:03.638207 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:04:03.638180    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/072b4622-e276-45df-80ae-b870afbbea26-trusted-ca-bundle\") pod \"072b4622-e276-45df-80ae-b870afbbea26\" (UID: \"072b4622-e276-45df-80ae-b870afbbea26\") "
Jul 29 04:04:03.638347 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:04:03.638217    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/072b4622-e276-45df-80ae-b870afbbea26-console-oauth-config\") pod \"072b4622-e276-45df-80ae-b870afbbea26\" (UID: \"072b4622-e276-45df-80ae-b870afbbea26\") "
Jul 29 04:04:03.638347 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:04:03.638241    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/072b4622-e276-45df-80ae-b870afbbea26-console-serving-cert\") pod \"072b4622-e276-45df-80ae-b870afbbea26\" (UID: \"072b4622-e276-45df-80ae-b870afbbea26\") "
Jul 29 04:04:03.638347 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:04:03.638269    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/072b4622-e276-45df-80ae-b870afbbea26-console-config\") pod \"072b4622-e276-45df-80ae-b870afbbea26\" (UID: \"072b4622-e276-45df-80ae-b870afbbea26\") "
Jul 29 04:04:03.638347 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:04:03.638335    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/072b4622-e276-45df-80ae-b870afbbea26-service-ca\") pod \"072b4622-e276-45df-80ae-b870afbbea26\" (UID: \"072b4622-e276-45df-80ae-b870afbbea26\") "
Jul 29 04:04:03.638556 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:04:03.638367    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/072b4622-e276-45df-80ae-b870afbbea26-oauth-serving-cert\") pod \"072b4622-e276-45df-80ae-b870afbbea26\" (UID: \"072b4622-e276-45df-80ae-b870afbbea26\") "
Jul 29 04:04:03.638556 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:04:03.638430    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xf9k7\" (UniqueName: \"kubernetes.io/projected/072b4622-e276-45df-80ae-b870afbbea26-kube-api-access-xf9k7\") pod \"072b4622-e276-45df-80ae-b870afbbea26\" (UID: \"072b4622-e276-45df-80ae-b870afbbea26\") "
Jul 29 04:04:03.638661 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:04:03.638571    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/072b4622-e276-45df-80ae-b870afbbea26-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "072b4622-e276-45df-80ae-b870afbbea26" (UID: "072b4622-e276-45df-80ae-b870afbbea26"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 04:04:03.638742 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:04:03.638723    2570 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/072b4622-e276-45df-80ae-b870afbbea26-trusted-ca-bundle\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:04:03.638802 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:04:03.638721    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/072b4622-e276-45df-80ae-b870afbbea26-console-config" (OuterVolumeSpecName: "console-config") pod "072b4622-e276-45df-80ae-b870afbbea26" (UID: "072b4622-e276-45df-80ae-b870afbbea26"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 04:04:03.639194 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:04:03.639078    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/072b4622-e276-45df-80ae-b870afbbea26-service-ca" (OuterVolumeSpecName: "service-ca") pod "072b4622-e276-45df-80ae-b870afbbea26" (UID: "072b4622-e276-45df-80ae-b870afbbea26"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 04:04:03.639384 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:04:03.639275    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/072b4622-e276-45df-80ae-b870afbbea26-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "072b4622-e276-45df-80ae-b870afbbea26" (UID: "072b4622-e276-45df-80ae-b870afbbea26"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 04:04:03.640548 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:04:03.640520    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/072b4622-e276-45df-80ae-b870afbbea26-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "072b4622-e276-45df-80ae-b870afbbea26" (UID: "072b4622-e276-45df-80ae-b870afbbea26"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 04:04:03.640640 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:04:03.640551    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/072b4622-e276-45df-80ae-b870afbbea26-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "072b4622-e276-45df-80ae-b870afbbea26" (UID: "072b4622-e276-45df-80ae-b870afbbea26"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 04:04:03.640684 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:04:03.640634    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/072b4622-e276-45df-80ae-b870afbbea26-kube-api-access-xf9k7" (OuterVolumeSpecName: "kube-api-access-xf9k7") pod "072b4622-e276-45df-80ae-b870afbbea26" (UID: "072b4622-e276-45df-80ae-b870afbbea26"). InnerVolumeSpecName "kube-api-access-xf9k7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 04:04:03.739173 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:04:03.739140    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xf9k7\" (UniqueName: \"kubernetes.io/projected/072b4622-e276-45df-80ae-b870afbbea26-kube-api-access-xf9k7\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:04:03.739173 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:04:03.739166    2570 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/072b4622-e276-45df-80ae-b870afbbea26-console-oauth-config\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:04:03.739339 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:04:03.739180    2570 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/072b4622-e276-45df-80ae-b870afbbea26-console-serving-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:04:03.739339 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:04:03.739192    2570 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/072b4622-e276-45df-80ae-b870afbbea26-console-config\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:04:03.739339 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:04:03.739204    2570 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/072b4622-e276-45df-80ae-b870afbbea26-service-ca\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:04:03.739339 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:04:03.739215    2570 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/072b4622-e276-45df-80ae-b870afbbea26-oauth-serving-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:04:04.349717 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:04:04.349685    2570 generic.go:356] "Generic (PLEG): container finished" podID="072b4622-e276-45df-80ae-b870afbbea26" containerID="f810ba5b7830329a9fb221ddbd547aa1f8d415638cfa57a3522bc4a33d43b3ec" exitCode=0
Jul 29 04:04:04.350164 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:04:04.349728    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-598469cf5b-ttwbp"
Jul 29 04:04:04.350164 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:04:04.349768    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-598469cf5b-ttwbp" event={"ID":"072b4622-e276-45df-80ae-b870afbbea26","Type":"ContainerDied","Data":"f810ba5b7830329a9fb221ddbd547aa1f8d415638cfa57a3522bc4a33d43b3ec"}
Jul 29 04:04:04.350164 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:04:04.349837    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-598469cf5b-ttwbp" event={"ID":"072b4622-e276-45df-80ae-b870afbbea26","Type":"ContainerDied","Data":"915c0093637e54544a968c76bed5ec3125739f19ef9c69befeb8d47798960bd6"}
Jul 29 04:04:04.350164 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:04:04.349857    2570 scope.go:117] "RemoveContainer" containerID="f810ba5b7830329a9fb221ddbd547aa1f8d415638cfa57a3522bc4a33d43b3ec"
Jul 29 04:04:04.360467 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:04:04.360450    2570 scope.go:117] "RemoveContainer" containerID="f810ba5b7830329a9fb221ddbd547aa1f8d415638cfa57a3522bc4a33d43b3ec"
Jul 29 04:04:04.360727 ip-10-0-135-138 kubenswrapper[2570]: E0729 04:04:04.360709    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f810ba5b7830329a9fb221ddbd547aa1f8d415638cfa57a3522bc4a33d43b3ec\": container with ID starting with f810ba5b7830329a9fb221ddbd547aa1f8d415638cfa57a3522bc4a33d43b3ec not found: ID does not exist" containerID="f810ba5b7830329a9fb221ddbd547aa1f8d415638cfa57a3522bc4a33d43b3ec"
Jul 29 04:04:04.360777 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:04:04.360735    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f810ba5b7830329a9fb221ddbd547aa1f8d415638cfa57a3522bc4a33d43b3ec"} err="failed to get container status \"f810ba5b7830329a9fb221ddbd547aa1f8d415638cfa57a3522bc4a33d43b3ec\": rpc error: code = NotFound desc = could not find container \"f810ba5b7830329a9fb221ddbd547aa1f8d415638cfa57a3522bc4a33d43b3ec\": container with ID starting with f810ba5b7830329a9fb221ddbd547aa1f8d415638cfa57a3522bc4a33d43b3ec not found: ID does not exist"
Jul 29 04:04:04.368971 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:04:04.368947    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-598469cf5b-ttwbp"]
Jul 29 04:04:04.373319 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:04:04.373298    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-598469cf5b-ttwbp"]
Jul 29 04:04:05.834971 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:04:05.834934    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="072b4622-e276-45df-80ae-b870afbbea26" path="/var/lib/kubelet/pods/072b4622-e276-45df-80ae-b870afbbea26/volumes"
Jul 29 04:04:27.384091 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:04:27.384012    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-4qq2r"]
Jul 29 04:04:27.384531 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:04:27.384399    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59fb94fa-8d20-42b7-8f94-136ff4a8d778" containerName="authorino"
Jul 29 04:04:27.384531 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:04:27.384410    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="59fb94fa-8d20-42b7-8f94-136ff4a8d778" containerName="authorino"
Jul 29 04:04:27.384531 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:04:27.384437    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="072b4622-e276-45df-80ae-b870afbbea26" containerName="console"
Jul 29 04:04:27.384531 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:04:27.384442    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="072b4622-e276-45df-80ae-b870afbbea26" containerName="console"
Jul 29 04:04:27.384531 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:04:27.384512    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="59fb94fa-8d20-42b7-8f94-136ff4a8d778" containerName="authorino"
Jul 29 04:04:27.384531 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:04:27.384527    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="072b4622-e276-45df-80ae-b870afbbea26" containerName="console"
Jul 29 04:04:27.387533 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:04:27.387517    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-4qq2r"
Jul 29 04:04:27.390530 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:04:27.390506    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 29 04:04:27.390640 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:04:27.390558    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-pbjrf\""
Jul 29 04:04:27.396212 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:04:27.396188    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-4qq2r"]
Jul 29 04:04:27.418418 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:04:27.418388    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-gznsm"]
Jul 29 04:04:27.421691 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:04:27.421676    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-gznsm"
Jul 29 04:04:27.434960 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:04:27.434931    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twsjz\" (UniqueName: \"kubernetes.io/projected/c237404f-9025-4bd0-9b69-b060923d2ef0-kube-api-access-twsjz\") pod \"payload-pre-processing-687dbb6cf6-4qq2r\" (UID: \"c237404f-9025-4bd0-9b69-b060923d2ef0\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-4qq2r"
Jul 29 04:04:27.435075 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:04:27.435053    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c237404f-9025-4bd0-9b69-b060923d2ef0-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-4qq2r\" (UID: \"c237404f-9025-4bd0-9b69-b060923d2ef0\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-4qq2r"
Jul 29 04:04:27.435135 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:04:27.435103    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1ffbe6e-a508-4235-a696-01cd077b7423-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-gznsm\" (UID: \"a1ffbe6e-a508-4235-a696-01cd077b7423\") " pod="openshift-ingress/payload-processing-5cc6cd444b-gznsm"
Jul 29 04:04:27.435189 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:04:27.435152    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htdcb\" (UniqueName: \"kubernetes.io/projected/a1ffbe6e-a508-4235-a696-01cd077b7423-kube-api-access-htdcb\") pod \"payload-processing-5cc6cd444b-gznsm\" (UID: \"a1ffbe6e-a508-4235-a696-01cd077b7423\") " pod="openshift-ingress/payload-processing-5cc6cd444b-gznsm"
Jul 29 04:04:27.436484 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:04:27.436465    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-gznsm"]
Jul 29 04:04:27.536430 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:04:27.536395    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c237404f-9025-4bd0-9b69-b060923d2ef0-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-4qq2r\" (UID: \"c237404f-9025-4bd0-9b69-b060923d2ef0\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-4qq2r"
Jul 29 04:04:27.536430 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:04:27.536436    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1ffbe6e-a508-4235-a696-01cd077b7423-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-gznsm\" (UID: \"a1ffbe6e-a508-4235-a696-01cd077b7423\") " pod="openshift-ingress/payload-processing-5cc6cd444b-gznsm"
Jul 29 04:04:27.536648 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:04:27.536524    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htdcb\" (UniqueName: \"kubernetes.io/projected/a1ffbe6e-a508-4235-a696-01cd077b7423-kube-api-access-htdcb\") pod \"payload-processing-5cc6cd444b-gznsm\" (UID: \"a1ffbe6e-a508-4235-a696-01cd077b7423\") " pod="openshift-ingress/payload-processing-5cc6cd444b-gznsm"
Jul 29 04:04:27.536648 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:04:27.536574    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twsjz\" (UniqueName: \"kubernetes.io/projected/c237404f-9025-4bd0-9b69-b060923d2ef0-kube-api-access-twsjz\") pod \"payload-pre-processing-687dbb6cf6-4qq2r\" (UID: \"c237404f-9025-4bd0-9b69-b060923d2ef0\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-4qq2r"
Jul 29 04:04:27.537288 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:04:27.537266    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1ffbe6e-a508-4235-a696-01cd077b7423-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-gznsm\" (UID: \"a1ffbe6e-a508-4235-a696-01cd077b7423\") " pod="openshift-ingress/payload-processing-5cc6cd444b-gznsm"
Jul 29 04:04:27.537378 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:04:27.537300    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c237404f-9025-4bd0-9b69-b060923d2ef0-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-4qq2r\" (UID: \"c237404f-9025-4bd0-9b69-b060923d2ef0\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-4qq2r"
Jul 29 04:04:27.546424 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:04:27.546396    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htdcb\" (UniqueName: \"kubernetes.io/projected/a1ffbe6e-a508-4235-a696-01cd077b7423-kube-api-access-htdcb\") pod \"payload-processing-5cc6cd444b-gznsm\" (UID: \"a1ffbe6e-a508-4235-a696-01cd077b7423\") " pod="openshift-ingress/payload-processing-5cc6cd444b-gznsm"
Jul 29 04:04:27.546627 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:04:27.546607    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twsjz\" (UniqueName: \"kubernetes.io/projected/c237404f-9025-4bd0-9b69-b060923d2ef0-kube-api-access-twsjz\") pod \"payload-pre-processing-687dbb6cf6-4qq2r\" (UID: \"c237404f-9025-4bd0-9b69-b060923d2ef0\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-4qq2r"
Jul 29 04:04:27.699645 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:04:27.699545    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-4qq2r"
Jul 29 04:04:27.730815 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:04:27.730789    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-gznsm"
Jul 29 04:04:27.850498 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:04:27.850468    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-4qq2r"]
Jul 29 04:04:27.852604 ip-10-0-135-138 kubenswrapper[2570]: W0729 04:04:27.852581    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc237404f_9025_4bd0_9b69_b060923d2ef0.slice/crio-54a93518ab30254453342cb1622e93020a0e78f634c878cfda435c8ad20d35ca WatchSource:0}: Error finding container 54a93518ab30254453342cb1622e93020a0e78f634c878cfda435c8ad20d35ca: Status 404 returned error can't find the container with id 54a93518ab30254453342cb1622e93020a0e78f634c878cfda435c8ad20d35ca
Jul 29 04:04:27.879068 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:04:27.879027    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-gznsm"]
Jul 29 04:04:27.882088 ip-10-0-135-138 kubenswrapper[2570]: W0729 04:04:27.882060    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1ffbe6e_a508_4235_a696_01cd077b7423.slice/crio-7bdaaae8bd7bc1295f14d85b17467dee08fbe79bec5e0a7e8b91a4692c8d4bf8 WatchSource:0}: Error finding container 7bdaaae8bd7bc1295f14d85b17467dee08fbe79bec5e0a7e8b91a4692c8d4bf8: Status 404 returned error can't find the container with id 7bdaaae8bd7bc1295f14d85b17467dee08fbe79bec5e0a7e8b91a4692c8d4bf8
Jul 29 04:04:28.468704 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:04:28.468668    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-gznsm" event={"ID":"a1ffbe6e-a508-4235-a696-01cd077b7423","Type":"ContainerStarted","Data":"7bdaaae8bd7bc1295f14d85b17467dee08fbe79bec5e0a7e8b91a4692c8d4bf8"}
Jul 29 04:04:28.469771 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:04:28.469747    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-4qq2r" event={"ID":"c237404f-9025-4bd0-9b69-b060923d2ef0","Type":"ContainerStarted","Data":"54a93518ab30254453342cb1622e93020a0e78f634c878cfda435c8ad20d35ca"}
Jul 29 04:04:30.481717 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:04:30.481680    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-4qq2r" event={"ID":"c237404f-9025-4bd0-9b69-b060923d2ef0","Type":"ContainerStarted","Data":"2d7aad1c8b9d7ce41722b753cf8652a3c4b3a62940ed6bf73bd8bdcaa8a5198c"}
Jul 29 04:04:30.482189 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:04:30.481859    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-4qq2r"
Jul 29 04:04:30.483166 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:04:30.483147    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-gznsm" event={"ID":"a1ffbe6e-a508-4235-a696-01cd077b7423","Type":"ContainerStarted","Data":"1e97c484d253a9eee0fec2aea200e53954d1a5ecba0d676fdf2e1f08441252f4"}
Jul 29 04:04:30.483235 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:04:30.483193    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-gznsm"
Jul 29 04:04:30.499418 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:04:30.499371    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-4qq2r" podStartSLOduration=1.473223822 podStartE2EDuration="3.499355291s" totalImagesPullingTime="2.026131469s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:04:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:04:30.499231739 +0000 UTC m=+535.229362823" watchObservedRunningTime="2026-07-29 04:04:30.499355291 +0000 UTC m=+535.229486362"
Jul 29 04:04:30.517216 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:04:30.517175    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-gznsm" podStartSLOduration=1.5302033800000001 podStartE2EDuration="3.517162784s" totalImagesPullingTime="1.986959404s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:04:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:04:30.516570719 +0000 UTC m=+535.246701794" watchObservedRunningTime="2026-07-29 04:04:30.517162784 +0000 UTC m=+535.247293854"
Jul 29 04:04:41.493140 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:04:41.493111    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-gznsm"
Jul 29 04:04:41.493612 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:04:41.493188    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-4qq2r"
Jul 29 04:05:35.955760 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:05:35.955720    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-lxxjr"]
Jul 29 04:05:35.959282 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:05:35.959259    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lxxjr"
Jul 29 04:05:35.963180 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:05:35.963163    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-h4cwp\""
Jul 29 04:05:35.963282 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:05:35.963165    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 29 04:05:35.964671 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:05:35.964652    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Jul 29 04:05:35.964851 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:05:35.964652    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 29 04:05:35.970844 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:05:35.970808    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-lxxjr"]
Jul 29 04:05:36.062866 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:05:36.062810    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44240c25-e95a-44d8-81c0-2b87772614c4-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lxxjr\" (UID: \"44240c25-e95a-44d8-81c0-2b87772614c4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lxxjr"
Jul 29 04:05:36.063029 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:05:36.062934    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44240c25-e95a-44d8-81c0-2b87772614c4-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lxxjr\" (UID: \"44240c25-e95a-44d8-81c0-2b87772614c4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lxxjr"
Jul 29 04:05:36.063029 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:05:36.062982    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44240c25-e95a-44d8-81c0-2b87772614c4-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lxxjr\" (UID: \"44240c25-e95a-44d8-81c0-2b87772614c4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lxxjr"
Jul 29 04:05:36.063029 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:05:36.063009    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44240c25-e95a-44d8-81c0-2b87772614c4-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lxxjr\" (UID: \"44240c25-e95a-44d8-81c0-2b87772614c4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lxxjr"
Jul 29 04:05:36.063149 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:05:36.063046    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6q6d\" (UniqueName: \"kubernetes.io/projected/44240c25-e95a-44d8-81c0-2b87772614c4-kube-api-access-g6q6d\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lxxjr\" (UID: \"44240c25-e95a-44d8-81c0-2b87772614c4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lxxjr"
Jul 29 04:05:36.063149 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:05:36.063075    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44240c25-e95a-44d8-81c0-2b87772614c4-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lxxjr\" (UID: \"44240c25-e95a-44d8-81c0-2b87772614c4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lxxjr"
Jul 29 04:05:36.163994 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:05:36.163964    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44240c25-e95a-44d8-81c0-2b87772614c4-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lxxjr\" (UID: \"44240c25-e95a-44d8-81c0-2b87772614c4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lxxjr"
Jul 29 04:05:36.164144 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:05:36.164003    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44240c25-e95a-44d8-81c0-2b87772614c4-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lxxjr\" (UID: \"44240c25-e95a-44d8-81c0-2b87772614c4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lxxjr"
Jul 29 04:05:36.164144 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:05:36.164041    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6q6d\" (UniqueName: \"kubernetes.io/projected/44240c25-e95a-44d8-81c0-2b87772614c4-kube-api-access-g6q6d\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lxxjr\" (UID: \"44240c25-e95a-44d8-81c0-2b87772614c4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lxxjr"
Jul 29 04:05:36.164144 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:05:36.164072    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44240c25-e95a-44d8-81c0-2b87772614c4-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lxxjr\" (UID: \"44240c25-e95a-44d8-81c0-2b87772614c4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lxxjr"
Jul 29 04:05:36.164144 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:05:36.164128    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44240c25-e95a-44d8-81c0-2b87772614c4-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lxxjr\" (UID: \"44240c25-e95a-44d8-81c0-2b87772614c4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lxxjr"
Jul 29 04:05:36.164358 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:05:36.164180    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44240c25-e95a-44d8-81c0-2b87772614c4-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lxxjr\" (UID: \"44240c25-e95a-44d8-81c0-2b87772614c4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lxxjr"
Jul 29 04:05:36.164414 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:05:36.164383    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44240c25-e95a-44d8-81c0-2b87772614c4-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lxxjr\" (UID: \"44240c25-e95a-44d8-81c0-2b87772614c4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lxxjr"
Jul 29 04:05:36.164464 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:05:36.164430    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44240c25-e95a-44d8-81c0-2b87772614c4-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lxxjr\" (UID: \"44240c25-e95a-44d8-81c0-2b87772614c4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lxxjr"
Jul 29 04:05:36.164517 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:05:36.164504    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44240c25-e95a-44d8-81c0-2b87772614c4-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lxxjr\" (UID: \"44240c25-e95a-44d8-81c0-2b87772614c4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lxxjr"
Jul 29 04:05:36.166514 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:05:36.166492    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44240c25-e95a-44d8-81c0-2b87772614c4-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lxxjr\" (UID: \"44240c25-e95a-44d8-81c0-2b87772614c4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lxxjr"
Jul 29 04:05:36.166767 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:05:36.166748    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44240c25-e95a-44d8-81c0-2b87772614c4-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lxxjr\" (UID: \"44240c25-e95a-44d8-81c0-2b87772614c4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lxxjr"
Jul 29 04:05:36.173363 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:05:36.173338    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6q6d\" (UniqueName: \"kubernetes.io/projected/44240c25-e95a-44d8-81c0-2b87772614c4-kube-api-access-g6q6d\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lxxjr\" (UID: \"44240c25-e95a-44d8-81c0-2b87772614c4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lxxjr"
Jul 29 04:05:36.270248 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:05:36.270174    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lxxjr"
Jul 29 04:05:36.403478 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:05:36.403456    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-lxxjr"]
Jul 29 04:05:36.405519 ip-10-0-135-138 kubenswrapper[2570]: W0729 04:05:36.405491    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44240c25_e95a_44d8_81c0_2b87772614c4.slice/crio-5e45a7a6521fd7c80eafeb9147d44fb290ec95849a4ee922d2b612ad14e8dbb4 WatchSource:0}: Error finding container 5e45a7a6521fd7c80eafeb9147d44fb290ec95849a4ee922d2b612ad14e8dbb4: Status 404 returned error can't find the container with id 5e45a7a6521fd7c80eafeb9147d44fb290ec95849a4ee922d2b612ad14e8dbb4
Jul 29 04:05:36.791794 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:05:36.791757    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lxxjr" event={"ID":"44240c25-e95a-44d8-81c0-2b87772614c4","Type":"ContainerStarted","Data":"5e45a7a6521fd7c80eafeb9147d44fb290ec95849a4ee922d2b612ad14e8dbb4"}
Jul 29 04:05:37.798464 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:05:37.798382    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lxxjr" event={"ID":"44240c25-e95a-44d8-81c0-2b87772614c4","Type":"ContainerStarted","Data":"e33b4c8c54552e28cac82b2fe496e16a1a1615127250fbb92acfca69f24be5c6"}
Jul 29 04:05:37.831469 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:05:37.831419    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lxxjr" podStartSLOduration=1.702664935 podStartE2EDuration="2.831405066s" totalImagesPullingTime="1.128740131s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:05:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:05:37.831100273 +0000 UTC m=+602.561231361" watchObservedRunningTime="2026-07-29 04:05:37.831405066 +0000 UTC m=+602.561536135"
Jul 29 04:05:46.270301 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:05:46.270263    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lxxjr"
Jul 29 04:05:46.272850 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:05:46.270332    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lxxjr"
Jul 29 04:05:46.283453 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:05:46.283428    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lxxjr"
Jul 29 04:05:46.852160 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:05:46.852130    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lxxjr"
Jul 29 04:05:52.029503 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:05:52.029469    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs87wf"]
Jul 29 04:05:52.032589 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:05:52.032570    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs87wf"
Jul 29 04:05:52.035599 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:05:52.035578    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Jul 29 04:05:52.044492 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:05:52.044467    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs87wf"]
Jul 29 04:05:52.104765 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:05:52.104729    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc0069b5-b84c-48de-80ab-86cf9718cc0b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs87wf\" (UID: \"cc0069b5-b84c-48de-80ab-86cf9718cc0b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs87wf"
Jul 29 04:05:52.104765 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:05:52.104765    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc0069b5-b84c-48de-80ab-86cf9718cc0b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs87wf\" (UID: \"cc0069b5-b84c-48de-80ab-86cf9718cc0b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs87wf"
Jul 29 04:05:52.104998 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:05:52.104792    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghnmf\" (UniqueName: \"kubernetes.io/projected/cc0069b5-b84c-48de-80ab-86cf9718cc0b-kube-api-access-ghnmf\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs87wf\" (UID: \"cc0069b5-b84c-48de-80ab-86cf9718cc0b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs87wf"
Jul 29 04:05:52.104998 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:05:52.104850    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc0069b5-b84c-48de-80ab-86cf9718cc0b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs87wf\" (UID: \"cc0069b5-b84c-48de-80ab-86cf9718cc0b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs87wf"
Jul 29 04:05:52.104998 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:05:52.104922    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc0069b5-b84c-48de-80ab-86cf9718cc0b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs87wf\" (UID: \"cc0069b5-b84c-48de-80ab-86cf9718cc0b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs87wf"
Jul 29 04:05:52.104998 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:05:52.104981    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc0069b5-b84c-48de-80ab-86cf9718cc0b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs87wf\" (UID: \"cc0069b5-b84c-48de-80ab-86cf9718cc0b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs87wf"
Jul 29 04:05:52.206050 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:05:52.206018    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc0069b5-b84c-48de-80ab-86cf9718cc0b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs87wf\" (UID: \"cc0069b5-b84c-48de-80ab-86cf9718cc0b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs87wf"
Jul 29 04:05:52.206050 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:05:52.206053    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc0069b5-b84c-48de-80ab-86cf9718cc0b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs87wf\" (UID: \"cc0069b5-b84c-48de-80ab-86cf9718cc0b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs87wf"
Jul 29 04:05:52.206293 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:05:52.206070    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghnmf\" (UniqueName: \"kubernetes.io/projected/cc0069b5-b84c-48de-80ab-86cf9718cc0b-kube-api-access-ghnmf\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs87wf\" (UID: \"cc0069b5-b84c-48de-80ab-86cf9718cc0b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs87wf"
Jul 29 04:05:52.206293 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:05:52.206104    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc0069b5-b84c-48de-80ab-86cf9718cc0b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs87wf\" (UID: \"cc0069b5-b84c-48de-80ab-86cf9718cc0b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs87wf"
Jul 29 04:05:52.206293 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:05:52.206139    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc0069b5-b84c-48de-80ab-86cf9718cc0b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs87wf\" (UID: \"cc0069b5-b84c-48de-80ab-86cf9718cc0b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs87wf"
Jul 29 04:05:52.206293 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:05:52.206192    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc0069b5-b84c-48de-80ab-86cf9718cc0b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs87wf\" (UID: \"cc0069b5-b84c-48de-80ab-86cf9718cc0b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs87wf"
Jul 29 04:05:52.206574 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:05:52.206549    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc0069b5-b84c-48de-80ab-86cf9718cc0b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs87wf\" (UID: \"cc0069b5-b84c-48de-80ab-86cf9718cc0b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs87wf"
Jul 29 04:05:52.206574 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:05:52.206566    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc0069b5-b84c-48de-80ab-86cf9718cc0b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs87wf\" (UID: \"cc0069b5-b84c-48de-80ab-86cf9718cc0b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs87wf"
Jul 29 04:05:52.206658 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:05:52.206631    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc0069b5-b84c-48de-80ab-86cf9718cc0b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs87wf\" (UID: \"cc0069b5-b84c-48de-80ab-86cf9718cc0b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs87wf"
Jul 29 04:05:52.208455 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:05:52.208436    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc0069b5-b84c-48de-80ab-86cf9718cc0b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs87wf\" (UID: \"cc0069b5-b84c-48de-80ab-86cf9718cc0b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs87wf"
Jul 29 04:05:52.209063 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:05:52.209047    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc0069b5-b84c-48de-80ab-86cf9718cc0b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs87wf\" (UID: \"cc0069b5-b84c-48de-80ab-86cf9718cc0b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs87wf"
Jul 29 04:05:52.214404 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:05:52.214381    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghnmf\" (UniqueName: \"kubernetes.io/projected/cc0069b5-b84c-48de-80ab-86cf9718cc0b-kube-api-access-ghnmf\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs87wf\" (UID: \"cc0069b5-b84c-48de-80ab-86cf9718cc0b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs87wf"
Jul 29 04:05:52.345556 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:05:52.345531    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs87wf"
Jul 29 04:05:52.480692 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:05:52.480667    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs87wf"]
Jul 29 04:05:52.482699 ip-10-0-135-138 kubenswrapper[2570]: W0729 04:05:52.482663    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc0069b5_b84c_48de_80ab_86cf9718cc0b.slice/crio-3212cd48b120c4fed39e6deb2db3a6fcdcb0bf6d6e5a3bb9bed08b31619fff2c WatchSource:0}: Error finding container 3212cd48b120c4fed39e6deb2db3a6fcdcb0bf6d6e5a3bb9bed08b31619fff2c: Status 404 returned error can't find the container with id 3212cd48b120c4fed39e6deb2db3a6fcdcb0bf6d6e5a3bb9bed08b31619fff2c
Jul 29 04:05:52.484633 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:05:52.484612    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 04:05:52.870490 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:05:52.870453    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs87wf" event={"ID":"cc0069b5-b84c-48de-80ab-86cf9718cc0b","Type":"ContainerStarted","Data":"b8eeb528ac156597f93d7e226104333c068d50651cebcde60e6d7589008e8929"}
Jul 29 04:05:52.870682 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:05:52.870508    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs87wf" event={"ID":"cc0069b5-b84c-48de-80ab-86cf9718cc0b","Type":"ContainerStarted","Data":"3212cd48b120c4fed39e6deb2db3a6fcdcb0bf6d6e5a3bb9bed08b31619fff2c"}
Jul 29 04:05:52.893320 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:05:52.893228    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs87wf" podStartSLOduration=0.734077703 podStartE2EDuration="893.212482ms" totalImagesPullingTime="159.134779ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:05:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:05:52.89140315 +0000 UTC m=+617.621534218" watchObservedRunningTime="2026-07-29 04:05:52.893212482 +0000 UTC m=+617.623343551"
Jul 29 04:06:02.346347 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:06:02.346263    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs87wf"
Jul 29 04:06:02.346347 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:06:02.346321    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs87wf"
Jul 29 04:06:02.358713 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:06:02.358690    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs87wf"
Jul 29 04:06:02.926426 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:06:02.926396    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs87wf"
Jul 29 04:06:15.031664 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:06:15.031616    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-zdg6k"]
Jul 29 04:06:15.038718 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:06:15.038699    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zdg6k"
Jul 29 04:06:15.042975 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:06:15.042954    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 29 04:06:15.045442 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:06:15.045416    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-zdg6k"]
Jul 29 04:06:15.107918 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:06:15.107880    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f64374e-db22-4963-bdf5-0ffce977a69e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zdg6k\" (UID: \"7f64374e-db22-4963-bdf5-0ffce977a69e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zdg6k"
Jul 29 04:06:15.108064 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:06:15.107941    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdxqz\" (UniqueName: \"kubernetes.io/projected/7f64374e-db22-4963-bdf5-0ffce977a69e-kube-api-access-kdxqz\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zdg6k\" (UID: \"7f64374e-db22-4963-bdf5-0ffce977a69e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zdg6k"
Jul 29 04:06:15.108064 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:06:15.108009    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f64374e-db22-4963-bdf5-0ffce977a69e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zdg6k\" (UID: \"7f64374e-db22-4963-bdf5-0ffce977a69e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zdg6k"
Jul 29 04:06:15.108064 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:06:15.108056    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f64374e-db22-4963-bdf5-0ffce977a69e-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zdg6k\" (UID: \"7f64374e-db22-4963-bdf5-0ffce977a69e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zdg6k"
Jul 29 04:06:15.108240 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:06:15.108091    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f64374e-db22-4963-bdf5-0ffce977a69e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zdg6k\" (UID: \"7f64374e-db22-4963-bdf5-0ffce977a69e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zdg6k"
Jul 29 04:06:15.108240 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:06:15.108191    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f64374e-db22-4963-bdf5-0ffce977a69e-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zdg6k\" (UID: \"7f64374e-db22-4963-bdf5-0ffce977a69e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zdg6k"
Jul 29 04:06:15.208880 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:06:15.208848    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f64374e-db22-4963-bdf5-0ffce977a69e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zdg6k\" (UID: \"7f64374e-db22-4963-bdf5-0ffce977a69e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zdg6k"
Jul 29 04:06:15.208880 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:06:15.208887    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f64374e-db22-4963-bdf5-0ffce977a69e-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zdg6k\" (UID: \"7f64374e-db22-4963-bdf5-0ffce977a69e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zdg6k"
Jul 29 04:06:15.209128 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:06:15.208909    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f64374e-db22-4963-bdf5-0ffce977a69e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zdg6k\" (UID: \"7f64374e-db22-4963-bdf5-0ffce977a69e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zdg6k"
Jul 29 04:06:15.209128 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:06:15.208970    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f64374e-db22-4963-bdf5-0ffce977a69e-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zdg6k\" (UID: \"7f64374e-db22-4963-bdf5-0ffce977a69e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zdg6k"
Jul 29 04:06:15.209128 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:06:15.208987    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f64374e-db22-4963-bdf5-0ffce977a69e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zdg6k\" (UID: \"7f64374e-db22-4963-bdf5-0ffce977a69e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zdg6k"
Jul 29 04:06:15.209128 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:06:15.209013    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdxqz\" (UniqueName: \"kubernetes.io/projected/7f64374e-db22-4963-bdf5-0ffce977a69e-kube-api-access-kdxqz\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zdg6k\" (UID: \"7f64374e-db22-4963-bdf5-0ffce977a69e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zdg6k"
Jul 29 04:06:15.209456 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:06:15.209425    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f64374e-db22-4963-bdf5-0ffce977a69e-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zdg6k\" (UID: \"7f64374e-db22-4963-bdf5-0ffce977a69e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zdg6k"
Jul 29 04:06:15.209793 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:06:15.209469    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f64374e-db22-4963-bdf5-0ffce977a69e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zdg6k\" (UID: \"7f64374e-db22-4963-bdf5-0ffce977a69e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zdg6k"
Jul 29 04:06:15.209899 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:06:15.209497    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f64374e-db22-4963-bdf5-0ffce977a69e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zdg6k\" (UID: \"7f64374e-db22-4963-bdf5-0ffce977a69e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zdg6k"
Jul 29 04:06:15.211219 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:06:15.211196    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f64374e-db22-4963-bdf5-0ffce977a69e-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zdg6k\" (UID: \"7f64374e-db22-4963-bdf5-0ffce977a69e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zdg6k"
Jul 29 04:06:15.211905 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:06:15.211886    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f64374e-db22-4963-bdf5-0ffce977a69e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zdg6k\" (UID: \"7f64374e-db22-4963-bdf5-0ffce977a69e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zdg6k"
Jul 29 04:06:15.220054 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:06:15.220035    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdxqz\" (UniqueName: \"kubernetes.io/projected/7f64374e-db22-4963-bdf5-0ffce977a69e-kube-api-access-kdxqz\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zdg6k\" (UID: \"7f64374e-db22-4963-bdf5-0ffce977a69e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zdg6k"
Jul 29 04:06:15.349236 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:06:15.349195    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zdg6k"
Jul 29 04:06:15.483489 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:06:15.483458    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-zdg6k"]
Jul 29 04:06:15.485002 ip-10-0-135-138 kubenswrapper[2570]: W0729 04:06:15.484975    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7f64374e_db22_4963_bdf5_0ffce977a69e.slice/crio-f4aa6cb331417aa92e2d15b8bc74a14d1b2f6b6260e685c72dab7139cc92d47e WatchSource:0}: Error finding container f4aa6cb331417aa92e2d15b8bc74a14d1b2f6b6260e685c72dab7139cc92d47e: Status 404 returned error can't find the container with id f4aa6cb331417aa92e2d15b8bc74a14d1b2f6b6260e685c72dab7139cc92d47e
Jul 29 04:06:15.976748 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:06:15.976657    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zdg6k" event={"ID":"7f64374e-db22-4963-bdf5-0ffce977a69e","Type":"ContainerStarted","Data":"b5d96b248275b6ee457e9b67a891d51cb5bc22135121ad0c32d53660453790f5"}
Jul 29 04:06:15.976748 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:06:15.976703    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zdg6k" event={"ID":"7f64374e-db22-4963-bdf5-0ffce977a69e","Type":"ContainerStarted","Data":"f4aa6cb331417aa92e2d15b8bc74a14d1b2f6b6260e685c72dab7139cc92d47e"}
Jul 29 04:06:15.996149 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:06:15.996103    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zdg6k" podStartSLOduration=0.82795552 podStartE2EDuration="996.086158ms" totalImagesPullingTime="168.130638ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:06:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:06:15.994323055 +0000 UTC m=+640.724454124" watchObservedRunningTime="2026-07-29 04:06:15.996086158 +0000 UTC m=+640.726217228"
Jul 29 04:06:25.349899 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:06:25.349860    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zdg6k"
Jul 29 04:06:25.350378 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:06:25.349914    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zdg6k"
Jul 29 04:06:25.362648 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:06:25.362614    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zdg6k"
Jul 29 04:06:26.035297 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:06:26.035271    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zdg6k"
Jul 29 04:16:05.285781 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:16:05.285743    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rxc52"]
Jul 29 04:16:05.286268 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:16:05.285960    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rxc52" podUID="6a0da2f7-c76c-4502-9977-be5904616395" containerName="manager" containerID="cri-o://ae4304475f5f9dfd9d039d0a322ab44702256f3c973b7fe3ff760784aff37164" gracePeriod=10
Jul 29 04:16:08.535539 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:16:08.535512    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rxc52"
Jul 29 04:16:08.653890 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:16:08.653788    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6a0da2f7-c76c-4502-9977-be5904616395-extensions-socket-volume\") pod \"6a0da2f7-c76c-4502-9977-be5904616395\" (UID: \"6a0da2f7-c76c-4502-9977-be5904616395\") "
Jul 29 04:16:08.653890 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:16:08.653883    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sq6mk\" (UniqueName: \"kubernetes.io/projected/6a0da2f7-c76c-4502-9977-be5904616395-kube-api-access-sq6mk\") pod \"6a0da2f7-c76c-4502-9977-be5904616395\" (UID: \"6a0da2f7-c76c-4502-9977-be5904616395\") "
Jul 29 04:16:08.654221 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:16:08.654198    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a0da2f7-c76c-4502-9977-be5904616395-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "6a0da2f7-c76c-4502-9977-be5904616395" (UID: "6a0da2f7-c76c-4502-9977-be5904616395"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 04:16:08.655957 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:16:08.655938    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a0da2f7-c76c-4502-9977-be5904616395-kube-api-access-sq6mk" (OuterVolumeSpecName: "kube-api-access-sq6mk") pod "6a0da2f7-c76c-4502-9977-be5904616395" (UID: "6a0da2f7-c76c-4502-9977-be5904616395"). InnerVolumeSpecName "kube-api-access-sq6mk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 04:16:08.755102 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:16:08.755069    2570 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6a0da2f7-c76c-4502-9977-be5904616395-extensions-socket-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:16:08.755102 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:16:08.755094    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sq6mk\" (UniqueName: \"kubernetes.io/projected/6a0da2f7-c76c-4502-9977-be5904616395-kube-api-access-sq6mk\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:16:08.785544 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:16:08.785507    2570 generic.go:356] "Generic (PLEG): container finished" podID="6a0da2f7-c76c-4502-9977-be5904616395" containerID="ae4304475f5f9dfd9d039d0a322ab44702256f3c973b7fe3ff760784aff37164" exitCode=0
Jul 29 04:16:08.785696 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:16:08.785552    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rxc52" event={"ID":"6a0da2f7-c76c-4502-9977-be5904616395","Type":"ContainerDied","Data":"ae4304475f5f9dfd9d039d0a322ab44702256f3c973b7fe3ff760784aff37164"}
Jul 29 04:16:08.785696 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:16:08.785593    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rxc52" event={"ID":"6a0da2f7-c76c-4502-9977-be5904616395","Type":"ContainerDied","Data":"4cf3ea1fdcac21fae2c9e2ecdf2febeb5558414d60b0334aeeef44daa6947cc1"}
Jul 29 04:16:08.785696 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:16:08.785614    2570 scope.go:117] "RemoveContainer" containerID="ae4304475f5f9dfd9d039d0a322ab44702256f3c973b7fe3ff760784aff37164"
Jul 29 04:16:08.785696 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:16:08.785555    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rxc52"
Jul 29 04:16:08.795919 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:16:08.795899    2570 scope.go:117] "RemoveContainer" containerID="ae4304475f5f9dfd9d039d0a322ab44702256f3c973b7fe3ff760784aff37164"
Jul 29 04:16:08.796180 ip-10-0-135-138 kubenswrapper[2570]: E0729 04:16:08.796160    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae4304475f5f9dfd9d039d0a322ab44702256f3c973b7fe3ff760784aff37164\": container with ID starting with ae4304475f5f9dfd9d039d0a322ab44702256f3c973b7fe3ff760784aff37164 not found: ID does not exist" containerID="ae4304475f5f9dfd9d039d0a322ab44702256f3c973b7fe3ff760784aff37164"
Jul 29 04:16:08.796218 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:16:08.796190    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae4304475f5f9dfd9d039d0a322ab44702256f3c973b7fe3ff760784aff37164"} err="failed to get container status \"ae4304475f5f9dfd9d039d0a322ab44702256f3c973b7fe3ff760784aff37164\": rpc error: code = NotFound desc = could not find container \"ae4304475f5f9dfd9d039d0a322ab44702256f3c973b7fe3ff760784aff37164\": container with ID starting with ae4304475f5f9dfd9d039d0a322ab44702256f3c973b7fe3ff760784aff37164 not found: ID does not exist"
Jul 29 04:16:08.808160 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:16:08.808137    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rxc52"]
Jul 29 04:16:08.812853 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:16:08.812834    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rxc52"]
Jul 29 04:16:09.834289 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:16:09.834259    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a0da2f7-c76c-4502-9977-be5904616395" path="/var/lib/kubelet/pods/6a0da2f7-c76c-4502-9977-be5904616395/volumes"
Jul 29 04:18:07.568928 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:18:07.568848    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-chq2s"]
Jul 29 04:18:07.569481 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:18:07.569455    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a0da2f7-c76c-4502-9977-be5904616395" containerName="manager"
Jul 29 04:18:07.569481 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:18:07.569474    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a0da2f7-c76c-4502-9977-be5904616395" containerName="manager"
Jul 29 04:18:07.569598 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:18:07.569573    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a0da2f7-c76c-4502-9977-be5904616395" containerName="manager"
Jul 29 04:18:07.572608 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:18:07.572588    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-chq2s"
Jul 29 04:18:07.575547 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:18:07.575524    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-pc9k8\""
Jul 29 04:18:07.575670 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:18:07.575530    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 29 04:18:07.575670 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:18:07.575530    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 29 04:18:07.593281 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:18:07.593256    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-chq2s"]
Jul 29 04:18:07.690962 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:18:07.690936    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c526\" (UniqueName: \"kubernetes.io/projected/9b6e291d-265a-4aa5-a74f-5763ee7294ae-kube-api-access-7c526\") pod \"kuadrant-operator-controller-manager-86878c769f-chq2s\" (UID: \"9b6e291d-265a-4aa5-a74f-5763ee7294ae\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-chq2s"
Jul 29 04:18:07.691109 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:18:07.691033    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9b6e291d-265a-4aa5-a74f-5763ee7294ae-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-chq2s\" (UID: \"9b6e291d-265a-4aa5-a74f-5763ee7294ae\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-chq2s"
Jul 29 04:18:07.792427 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:18:07.792397    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c526\" (UniqueName: \"kubernetes.io/projected/9b6e291d-265a-4aa5-a74f-5763ee7294ae-kube-api-access-7c526\") pod \"kuadrant-operator-controller-manager-86878c769f-chq2s\" (UID: \"9b6e291d-265a-4aa5-a74f-5763ee7294ae\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-chq2s"
Jul 29 04:18:07.792572 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:18:07.792480    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9b6e291d-265a-4aa5-a74f-5763ee7294ae-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-chq2s\" (UID: \"9b6e291d-265a-4aa5-a74f-5763ee7294ae\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-chq2s"
Jul 29 04:18:07.792872 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:18:07.792855    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9b6e291d-265a-4aa5-a74f-5763ee7294ae-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-chq2s\" (UID: \"9b6e291d-265a-4aa5-a74f-5763ee7294ae\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-chq2s"
Jul 29 04:18:07.800575 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:18:07.800549    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c526\" (UniqueName: \"kubernetes.io/projected/9b6e291d-265a-4aa5-a74f-5763ee7294ae-kube-api-access-7c526\") pod \"kuadrant-operator-controller-manager-86878c769f-chq2s\" (UID: \"9b6e291d-265a-4aa5-a74f-5763ee7294ae\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-chq2s"
Jul 29 04:18:07.882991 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:18:07.882963    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-chq2s"
Jul 29 04:18:08.019000 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:18:08.018978    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-chq2s"]
Jul 29 04:18:08.021713 ip-10-0-135-138 kubenswrapper[2570]: W0729 04:18:08.021687    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b6e291d_265a_4aa5_a74f_5763ee7294ae.slice/crio-4aaf293d03a3f5abcbfd5a8a82e61ca278f5359505412c358642521981024517 WatchSource:0}: Error finding container 4aaf293d03a3f5abcbfd5a8a82e61ca278f5359505412c358642521981024517: Status 404 returned error can't find the container with id 4aaf293d03a3f5abcbfd5a8a82e61ca278f5359505412c358642521981024517
Jul 29 04:18:08.339115 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:18:08.339085    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-chq2s" event={"ID":"9b6e291d-265a-4aa5-a74f-5763ee7294ae","Type":"ContainerStarted","Data":"cd81f34cbcf921d22645d6b586df7a16a23edf7cb0dec8e891f074a599733036"}
Jul 29 04:18:08.339290 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:18:08.339152    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-chq2s"
Jul 29 04:18:08.339290 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:18:08.339176    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-chq2s" event={"ID":"9b6e291d-265a-4aa5-a74f-5763ee7294ae","Type":"ContainerStarted","Data":"4aaf293d03a3f5abcbfd5a8a82e61ca278f5359505412c358642521981024517"}
Jul 29 04:18:08.359434 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:18:08.359392    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-chq2s" podStartSLOduration=1.359377652 podStartE2EDuration="1.359377652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:18:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:18:08.357331266 +0000 UTC m=+1353.087462334" watchObservedRunningTime="2026-07-29 04:18:08.359377652 +0000 UTC m=+1353.089508721"
Jul 29 04:18:19.345508 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:18:19.345472    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-chq2s"
Jul 29 04:21:05.927570 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:21:05.927477    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-0aedf407-kserve-5644dd669f-j6x7t"]
Jul 29 04:21:05.931682 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:21:05.931660    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-0aedf407-kserve-5644dd669f-j6x7t"
Jul 29 04:21:05.934182 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:21:05.934160    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-0aedf407-kserve-self-signed-certs\""
Jul 29 04:21:05.938948 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:21:05.938928    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-0aedf407-kserve-5644dd669f-j6x7t"]
Jul 29 04:21:05.983642 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:21:05.983612    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr948\" (UniqueName: \"kubernetes.io/projected/9814f924-be25-4694-b95d-ad32c72779ba-kube-api-access-wr948\") pod \"e2e-same-id-b-0aedf407-kserve-5644dd669f-j6x7t\" (UID: \"9814f924-be25-4694-b95d-ad32c72779ba\") " pod="llm/e2e-same-id-b-0aedf407-kserve-5644dd669f-j6x7t"
Jul 29 04:21:05.983772 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:21:05.983652    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9814f924-be25-4694-b95d-ad32c72779ba-tmp-dir\") pod \"e2e-same-id-b-0aedf407-kserve-5644dd669f-j6x7t\" (UID: \"9814f924-be25-4694-b95d-ad32c72779ba\") " pod="llm/e2e-same-id-b-0aedf407-kserve-5644dd669f-j6x7t"
Jul 29 04:21:05.983772 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:21:05.983683    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9814f924-be25-4694-b95d-ad32c72779ba-home\") pod \"e2e-same-id-b-0aedf407-kserve-5644dd669f-j6x7t\" (UID: \"9814f924-be25-4694-b95d-ad32c72779ba\") " pod="llm/e2e-same-id-b-0aedf407-kserve-5644dd669f-j6x7t"
Jul 29 04:21:05.983772 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:21:05.983707    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9814f924-be25-4694-b95d-ad32c72779ba-model-cache\") pod \"e2e-same-id-b-0aedf407-kserve-5644dd669f-j6x7t\" (UID: \"9814f924-be25-4694-b95d-ad32c72779ba\") " pod="llm/e2e-same-id-b-0aedf407-kserve-5644dd669f-j6x7t"
Jul 29 04:21:05.983772 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:21:05.983739    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9814f924-be25-4694-b95d-ad32c72779ba-dshm\") pod \"e2e-same-id-b-0aedf407-kserve-5644dd669f-j6x7t\" (UID: \"9814f924-be25-4694-b95d-ad32c72779ba\") " pod="llm/e2e-same-id-b-0aedf407-kserve-5644dd669f-j6x7t"
Jul 29 04:21:05.983772 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:21:05.983765    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9814f924-be25-4694-b95d-ad32c72779ba-tls-certs\") pod \"e2e-same-id-b-0aedf407-kserve-5644dd669f-j6x7t\" (UID: \"9814f924-be25-4694-b95d-ad32c72779ba\") " pod="llm/e2e-same-id-b-0aedf407-kserve-5644dd669f-j6x7t"
Jul 29 04:21:06.084897 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:21:06.084865    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9814f924-be25-4694-b95d-ad32c72779ba-tls-certs\") pod \"e2e-same-id-b-0aedf407-kserve-5644dd669f-j6x7t\" (UID: \"9814f924-be25-4694-b95d-ad32c72779ba\") " pod="llm/e2e-same-id-b-0aedf407-kserve-5644dd669f-j6x7t"
Jul 29 04:21:06.085039 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:21:06.084972    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr948\" (UniqueName: \"kubernetes.io/projected/9814f924-be25-4694-b95d-ad32c72779ba-kube-api-access-wr948\") pod \"e2e-same-id-b-0aedf407-kserve-5644dd669f-j6x7t\" (UID: \"9814f924-be25-4694-b95d-ad32c72779ba\") " pod="llm/e2e-same-id-b-0aedf407-kserve-5644dd669f-j6x7t"
Jul 29 04:21:06.085039 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:21:06.085013    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9814f924-be25-4694-b95d-ad32c72779ba-tmp-dir\") pod \"e2e-same-id-b-0aedf407-kserve-5644dd669f-j6x7t\" (UID: \"9814f924-be25-4694-b95d-ad32c72779ba\") " pod="llm/e2e-same-id-b-0aedf407-kserve-5644dd669f-j6x7t"
Jul 29 04:21:06.085152 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:21:06.085046    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9814f924-be25-4694-b95d-ad32c72779ba-home\") pod \"e2e-same-id-b-0aedf407-kserve-5644dd669f-j6x7t\" (UID: \"9814f924-be25-4694-b95d-ad32c72779ba\") " pod="llm/e2e-same-id-b-0aedf407-kserve-5644dd669f-j6x7t"
Jul 29 04:21:06.085152 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:21:06.085070    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9814f924-be25-4694-b95d-ad32c72779ba-model-cache\") pod \"e2e-same-id-b-0aedf407-kserve-5644dd669f-j6x7t\" (UID: \"9814f924-be25-4694-b95d-ad32c72779ba\") " pod="llm/e2e-same-id-b-0aedf407-kserve-5644dd669f-j6x7t"
Jul 29 04:21:06.085152 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:21:06.085106    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9814f924-be25-4694-b95d-ad32c72779ba-dshm\") pod \"e2e-same-id-b-0aedf407-kserve-5644dd669f-j6x7t\" (UID: \"9814f924-be25-4694-b95d-ad32c72779ba\") " pod="llm/e2e-same-id-b-0aedf407-kserve-5644dd669f-j6x7t"
Jul 29 04:21:06.085425 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:21:06.085401    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9814f924-be25-4694-b95d-ad32c72779ba-tmp-dir\") pod \"e2e-same-id-b-0aedf407-kserve-5644dd669f-j6x7t\" (UID: \"9814f924-be25-4694-b95d-ad32c72779ba\") " pod="llm/e2e-same-id-b-0aedf407-kserve-5644dd669f-j6x7t"
Jul 29 04:21:06.085480 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:21:06.085420    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9814f924-be25-4694-b95d-ad32c72779ba-model-cache\") pod \"e2e-same-id-b-0aedf407-kserve-5644dd669f-j6x7t\" (UID: \"9814f924-be25-4694-b95d-ad32c72779ba\") " pod="llm/e2e-same-id-b-0aedf407-kserve-5644dd669f-j6x7t"
Jul 29 04:21:06.085480 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:21:06.085461    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9814f924-be25-4694-b95d-ad32c72779ba-home\") pod \"e2e-same-id-b-0aedf407-kserve-5644dd669f-j6x7t\" (UID: \"9814f924-be25-4694-b95d-ad32c72779ba\") " pod="llm/e2e-same-id-b-0aedf407-kserve-5644dd669f-j6x7t"
Jul 29 04:21:06.087381 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:21:06.087357    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9814f924-be25-4694-b95d-ad32c72779ba-dshm\") pod \"e2e-same-id-b-0aedf407-kserve-5644dd669f-j6x7t\" (UID: \"9814f924-be25-4694-b95d-ad32c72779ba\") " pod="llm/e2e-same-id-b-0aedf407-kserve-5644dd669f-j6x7t"
Jul 29 04:21:06.087472 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:21:06.087387    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9814f924-be25-4694-b95d-ad32c72779ba-tls-certs\") pod \"e2e-same-id-b-0aedf407-kserve-5644dd669f-j6x7t\" (UID: \"9814f924-be25-4694-b95d-ad32c72779ba\") " pod="llm/e2e-same-id-b-0aedf407-kserve-5644dd669f-j6x7t"
Jul 29 04:21:06.092519 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:21:06.092501    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr948\" (UniqueName: \"kubernetes.io/projected/9814f924-be25-4694-b95d-ad32c72779ba-kube-api-access-wr948\") pod \"e2e-same-id-b-0aedf407-kserve-5644dd669f-j6x7t\" (UID: \"9814f924-be25-4694-b95d-ad32c72779ba\") " pod="llm/e2e-same-id-b-0aedf407-kserve-5644dd669f-j6x7t"
Jul 29 04:21:06.243334 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:21:06.243264    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-0aedf407-kserve-5644dd669f-j6x7t"
Jul 29 04:21:06.374665 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:21:06.374562    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-0aedf407-kserve-5644dd669f-j6x7t"]
Jul 29 04:21:06.377533 ip-10-0-135-138 kubenswrapper[2570]: W0729 04:21:06.377501    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9814f924_be25_4694_b95d_ad32c72779ba.slice/crio-246a5684bba85f96bb9d4053d79370c43eff8b5d7885f8c99e5ef86225539c14 WatchSource:0}: Error finding container 246a5684bba85f96bb9d4053d79370c43eff8b5d7885f8c99e5ef86225539c14: Status 404 returned error can't find the container with id 246a5684bba85f96bb9d4053d79370c43eff8b5d7885f8c99e5ef86225539c14
Jul 29 04:21:07.192577 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:21:07.192540    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-0aedf407-kserve-5644dd669f-j6x7t" event={"ID":"9814f924-be25-4694-b95d-ad32c72779ba","Type":"ContainerStarted","Data":"49719035ad4bfb5890a590888c6170d837d56424d23991ba9c49676c8ea02c92"}
Jul 29 04:21:07.192972 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:21:07.192584    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-0aedf407-kserve-5644dd669f-j6x7t" event={"ID":"9814f924-be25-4694-b95d-ad32c72779ba","Type":"ContainerStarted","Data":"246a5684bba85f96bb9d4053d79370c43eff8b5d7885f8c99e5ef86225539c14"}
Jul 29 04:21:07.209282 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:21:07.209240    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-0aedf407-kserve-5644dd669f-j6x7t" podStartSLOduration=2.2092259260000002 podStartE2EDuration="2.209225926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:21:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:21:07.20765289 +0000 UTC m=+1531.937783965" watchObservedRunningTime="2026-07-29 04:21:07.209225926 +0000 UTC m=+1531.939357000"
Jul 29 04:21:16.244377 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:21:16.244349    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-0aedf407-kserve-5644dd669f-j6x7t"
Jul 29 04:21:16.244841 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:21:16.244407    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-0aedf407-kserve-5644dd669f-j6x7t"
Jul 29 04:21:16.256923 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:21:16.256904    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-0aedf407-kserve-5644dd669f-j6x7t"
Jul 29 04:21:17.259777 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:21:17.259751    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-0aedf407-kserve-5644dd669f-j6x7t"
Jul 29 04:21:53.784089 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:21:53.784056    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-0aedf407-kserve-5644dd669f-j6x7t"]
Jul 29 04:22:08.804997 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:22:08.804955    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-0aedf407-kserve-5644dd669f-j6x7t" podUID="9814f924-be25-4694-b95d-ad32c72779ba" containerName="main" containerID="cri-o://49719035ad4bfb5890a590888c6170d837d56424d23991ba9c49676c8ea02c92" gracePeriod=45
Jul 29 04:22:09.047251 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:22:09.047229    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-0aedf407-kserve-5644dd669f-j6x7t"
Jul 29 04:22:09.151778 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:22:09.151753    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9814f924-be25-4694-b95d-ad32c72779ba-dshm\") pod \"9814f924-be25-4694-b95d-ad32c72779ba\" (UID: \"9814f924-be25-4694-b95d-ad32c72779ba\") "
Jul 29 04:22:09.151950 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:22:09.151890    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9814f924-be25-4694-b95d-ad32c72779ba-home\") pod \"9814f924-be25-4694-b95d-ad32c72779ba\" (UID: \"9814f924-be25-4694-b95d-ad32c72779ba\") "
Jul 29 04:22:09.151950 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:22:09.151929    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9814f924-be25-4694-b95d-ad32c72779ba-model-cache\") pod \"9814f924-be25-4694-b95d-ad32c72779ba\" (UID: \"9814f924-be25-4694-b95d-ad32c72779ba\") "
Jul 29 04:22:09.152031 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:22:09.151981    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9814f924-be25-4694-b95d-ad32c72779ba-tmp-dir\") pod \"9814f924-be25-4694-b95d-ad32c72779ba\" (UID: \"9814f924-be25-4694-b95d-ad32c72779ba\") "
Jul 29 04:22:09.152031 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:22:09.152014    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9814f924-be25-4694-b95d-ad32c72779ba-tls-certs\") pod \"9814f924-be25-4694-b95d-ad32c72779ba\" (UID: \"9814f924-be25-4694-b95d-ad32c72779ba\") "
Jul 29 04:22:09.152129 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:22:09.152055    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wr948\" (UniqueName: \"kubernetes.io/projected/9814f924-be25-4694-b95d-ad32c72779ba-kube-api-access-wr948\") pod \"9814f924-be25-4694-b95d-ad32c72779ba\" (UID: \"9814f924-be25-4694-b95d-ad32c72779ba\") "
Jul 29 04:22:09.152187 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:22:09.152144    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9814f924-be25-4694-b95d-ad32c72779ba-home" (OuterVolumeSpecName: "home") pod "9814f924-be25-4694-b95d-ad32c72779ba" (UID: "9814f924-be25-4694-b95d-ad32c72779ba"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 04:22:09.152272 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:22:09.152248    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9814f924-be25-4694-b95d-ad32c72779ba-model-cache" (OuterVolumeSpecName: "model-cache") pod "9814f924-be25-4694-b95d-ad32c72779ba" (UID: "9814f924-be25-4694-b95d-ad32c72779ba"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 04:22:09.152335 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:22:09.152262    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9814f924-be25-4694-b95d-ad32c72779ba-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9814f924-be25-4694-b95d-ad32c72779ba" (UID: "9814f924-be25-4694-b95d-ad32c72779ba"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 04:22:09.152441 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:22:09.152425    2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9814f924-be25-4694-b95d-ad32c72779ba-home\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:22:09.152495 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:22:09.152443    2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9814f924-be25-4694-b95d-ad32c72779ba-model-cache\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:22:09.152495 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:22:09.152453    2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9814f924-be25-4694-b95d-ad32c72779ba-tmp-dir\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:22:09.154245 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:22:09.154222    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9814f924-be25-4694-b95d-ad32c72779ba-dshm" (OuterVolumeSpecName: "dshm") pod "9814f924-be25-4694-b95d-ad32c72779ba" (UID: "9814f924-be25-4694-b95d-ad32c72779ba"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 04:22:09.154363 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:22:09.154276    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9814f924-be25-4694-b95d-ad32c72779ba-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9814f924-be25-4694-b95d-ad32c72779ba" (UID: "9814f924-be25-4694-b95d-ad32c72779ba"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 04:22:09.154363 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:22:09.154288    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9814f924-be25-4694-b95d-ad32c72779ba-kube-api-access-wr948" (OuterVolumeSpecName: "kube-api-access-wr948") pod "9814f924-be25-4694-b95d-ad32c72779ba" (UID: "9814f924-be25-4694-b95d-ad32c72779ba"). InnerVolumeSpecName "kube-api-access-wr948". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 04:22:09.253500 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:22:09.253467    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9814f924-be25-4694-b95d-ad32c72779ba-tls-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:22:09.253500 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:22:09.253495    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wr948\" (UniqueName: \"kubernetes.io/projected/9814f924-be25-4694-b95d-ad32c72779ba-kube-api-access-wr948\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:22:09.253500 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:22:09.253505    2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9814f924-be25-4694-b95d-ad32c72779ba-dshm\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:22:09.498949 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:22:09.498867    2570 generic.go:356] "Generic (PLEG): container finished" podID="9814f924-be25-4694-b95d-ad32c72779ba" containerID="49719035ad4bfb5890a590888c6170d837d56424d23991ba9c49676c8ea02c92" exitCode=0
Jul 29 04:22:09.498949 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:22:09.498913    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-0aedf407-kserve-5644dd669f-j6x7t"
Jul 29 04:22:09.499159 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:22:09.498956    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-0aedf407-kserve-5644dd669f-j6x7t" event={"ID":"9814f924-be25-4694-b95d-ad32c72779ba","Type":"ContainerDied","Data":"49719035ad4bfb5890a590888c6170d837d56424d23991ba9c49676c8ea02c92"}
Jul 29 04:22:09.499159 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:22:09.499003    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-0aedf407-kserve-5644dd669f-j6x7t" event={"ID":"9814f924-be25-4694-b95d-ad32c72779ba","Type":"ContainerDied","Data":"246a5684bba85f96bb9d4053d79370c43eff8b5d7885f8c99e5ef86225539c14"}
Jul 29 04:22:09.499159 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:22:09.499021    2570 scope.go:117] "RemoveContainer" containerID="49719035ad4bfb5890a590888c6170d837d56424d23991ba9c49676c8ea02c92"
Jul 29 04:22:09.508173 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:22:09.508154    2570 scope.go:117] "RemoveContainer" containerID="49719035ad4bfb5890a590888c6170d837d56424d23991ba9c49676c8ea02c92"
Jul 29 04:22:09.516143 ip-10-0-135-138 kubenswrapper[2570]: E0729 04:22:09.516118    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49719035ad4bfb5890a590888c6170d837d56424d23991ba9c49676c8ea02c92\": container with ID starting with 49719035ad4bfb5890a590888c6170d837d56424d23991ba9c49676c8ea02c92 not found: ID does not exist" containerID="49719035ad4bfb5890a590888c6170d837d56424d23991ba9c49676c8ea02c92"
Jul 29 04:22:09.516266 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:22:09.516160    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49719035ad4bfb5890a590888c6170d837d56424d23991ba9c49676c8ea02c92"} err="failed to get container status \"49719035ad4bfb5890a590888c6170d837d56424d23991ba9c49676c8ea02c92\": rpc error: code = NotFound desc = could not find container \"49719035ad4bfb5890a590888c6170d837d56424d23991ba9c49676c8ea02c92\": container with ID starting with 49719035ad4bfb5890a590888c6170d837d56424d23991ba9c49676c8ea02c92 not found: ID does not exist"
Jul 29 04:22:09.520546 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:22:09.520523    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-0aedf407-kserve-5644dd669f-j6x7t"]
Jul 29 04:22:09.526356 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:22:09.526337    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-0aedf407-kserve-5644dd669f-j6x7t"]
Jul 29 04:22:09.834652 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:22:09.834620    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9814f924-be25-4694-b95d-ad32c72779ba" path="/var/lib/kubelet/pods/9814f924-be25-4694-b95d-ad32c72779ba/volumes"
Jul 29 04:26:44.088603 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:26:44.088568    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dd29cdf8-5b4d48784-rb9rv"]
Jul 29 04:26:44.089093 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:26:44.088976    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9814f924-be25-4694-b95d-ad32c72779ba" containerName="main"
Jul 29 04:26:44.089093 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:26:44.088987    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="9814f924-be25-4694-b95d-ad32c72779ba" containerName="main"
Jul 29 04:26:44.089093 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:26:44.089050    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="9814f924-be25-4694-b95d-ad32c72779ba" containerName="main"
Jul 29 04:26:44.091961 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:26:44.091943    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-dd29cdf8-5b4d48784-rb9rv"
Jul 29 04:26:44.096473 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:26:44.096454    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-dd29cdf8-dockercfg-2nzff\""
Jul 29 04:26:44.097032 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:26:44.097013    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-dd29cdf8\""
Jul 29 04:26:44.105489 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:26:44.105464    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dd29cdf8-5b4d48784-rb9rv"]
Jul 29 04:26:44.120561 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:26:44.120533    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dd29cdf8-59b7ddcf44-9lsnc"]
Jul 29 04:26:44.124086 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:26:44.124065    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-dd29cdf8-59b7ddcf44-9lsnc"
Jul 29 04:26:44.132706 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:26:44.132680    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dd29cdf8-59b7ddcf44-9lsnc"]
Jul 29 04:26:44.249276 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:26:44.249246    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21a6f7e6-e3d7-410b-a737-f223d4f9d76b-plugins-config-volume\") pod \"payload-processing-e2e-ait-dd29cdf8-59b7ddcf44-9lsnc\" (UID: \"21a6f7e6-e3d7-410b-a737-f223d4f9d76b\") " pod="openshift-ingress/payload-processing-e2e-ait-dd29cdf8-59b7ddcf44-9lsnc"
Jul 29 04:26:44.249444 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:26:44.249288    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjjch\" (UniqueName: \"kubernetes.io/projected/21a6f7e6-e3d7-410b-a737-f223d4f9d76b-kube-api-access-vjjch\") pod \"payload-processing-e2e-ait-dd29cdf8-59b7ddcf44-9lsnc\" (UID: \"21a6f7e6-e3d7-410b-a737-f223d4f9d76b\") " pod="openshift-ingress/payload-processing-e2e-ait-dd29cdf8-59b7ddcf44-9lsnc"
Jul 29 04:26:44.249444 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:26:44.249333    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/325af13c-07c6-47b6-8176-ef1567ed68b7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-dd29cdf8-5b4d48784-rb9rv\" (UID: \"325af13c-07c6-47b6-8176-ef1567ed68b7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dd29cdf8-5b4d48784-rb9rv"
Jul 29 04:26:44.249444 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:26:44.249388    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76fn8\" (UniqueName: \"kubernetes.io/projected/325af13c-07c6-47b6-8176-ef1567ed68b7-kube-api-access-76fn8\") pod \"payload-pre-processing-e2e-ait-dd29cdf8-5b4d48784-rb9rv\" (UID: \"325af13c-07c6-47b6-8176-ef1567ed68b7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dd29cdf8-5b4d48784-rb9rv"
Jul 29 04:26:44.350410 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:26:44.350323    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/325af13c-07c6-47b6-8176-ef1567ed68b7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-dd29cdf8-5b4d48784-rb9rv\" (UID: \"325af13c-07c6-47b6-8176-ef1567ed68b7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dd29cdf8-5b4d48784-rb9rv"
Jul 29 04:26:44.350410 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:26:44.350382    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76fn8\" (UniqueName: \"kubernetes.io/projected/325af13c-07c6-47b6-8176-ef1567ed68b7-kube-api-access-76fn8\") pod \"payload-pre-processing-e2e-ait-dd29cdf8-5b4d48784-rb9rv\" (UID: \"325af13c-07c6-47b6-8176-ef1567ed68b7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dd29cdf8-5b4d48784-rb9rv"
Jul 29 04:26:44.350640 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:26:44.350422    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21a6f7e6-e3d7-410b-a737-f223d4f9d76b-plugins-config-volume\") pod \"payload-processing-e2e-ait-dd29cdf8-59b7ddcf44-9lsnc\" (UID: \"21a6f7e6-e3d7-410b-a737-f223d4f9d76b\") " pod="openshift-ingress/payload-processing-e2e-ait-dd29cdf8-59b7ddcf44-9lsnc"
Jul 29 04:26:44.350640 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:26:44.350448    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjjch\" (UniqueName: \"kubernetes.io/projected/21a6f7e6-e3d7-410b-a737-f223d4f9d76b-kube-api-access-vjjch\") pod \"payload-processing-e2e-ait-dd29cdf8-59b7ddcf44-9lsnc\" (UID: \"21a6f7e6-e3d7-410b-a737-f223d4f9d76b\") " pod="openshift-ingress/payload-processing-e2e-ait-dd29cdf8-59b7ddcf44-9lsnc"
Jul 29 04:26:44.351318 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:26:44.351295    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21a6f7e6-e3d7-410b-a737-f223d4f9d76b-plugins-config-volume\") pod \"payload-processing-e2e-ait-dd29cdf8-59b7ddcf44-9lsnc\" (UID: \"21a6f7e6-e3d7-410b-a737-f223d4f9d76b\") " pod="openshift-ingress/payload-processing-e2e-ait-dd29cdf8-59b7ddcf44-9lsnc"
Jul 29 04:26:44.351361 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:26:44.351322    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/325af13c-07c6-47b6-8176-ef1567ed68b7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-dd29cdf8-5b4d48784-rb9rv\" (UID: \"325af13c-07c6-47b6-8176-ef1567ed68b7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dd29cdf8-5b4d48784-rb9rv"
Jul 29 04:26:44.358813 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:26:44.358789    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjjch\" (UniqueName: \"kubernetes.io/projected/21a6f7e6-e3d7-410b-a737-f223d4f9d76b-kube-api-access-vjjch\") pod \"payload-processing-e2e-ait-dd29cdf8-59b7ddcf44-9lsnc\" (UID: \"21a6f7e6-e3d7-410b-a737-f223d4f9d76b\") " pod="openshift-ingress/payload-processing-e2e-ait-dd29cdf8-59b7ddcf44-9lsnc"
Jul 29 04:26:44.359039 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:26:44.359018    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76fn8\" (UniqueName: \"kubernetes.io/projected/325af13c-07c6-47b6-8176-ef1567ed68b7-kube-api-access-76fn8\") pod \"payload-pre-processing-e2e-ait-dd29cdf8-5b4d48784-rb9rv\" (UID: \"325af13c-07c6-47b6-8176-ef1567ed68b7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dd29cdf8-5b4d48784-rb9rv"
Jul 29 04:26:44.402527 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:26:44.402495    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-dd29cdf8-5b4d48784-rb9rv"
Jul 29 04:26:44.435458 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:26:44.435426    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-dd29cdf8-59b7ddcf44-9lsnc"
Jul 29 04:26:44.546502 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:26:44.546477    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dd29cdf8-5b4d48784-rb9rv"]
Jul 29 04:26:44.549488 ip-10-0-135-138 kubenswrapper[2570]: W0729 04:26:44.549313    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod325af13c_07c6_47b6_8176_ef1567ed68b7.slice/crio-f0e4d9d6ae4dfcc0dc4af0b30f265c090dbeacbf2208bc70dcfe8240828e80d3 WatchSource:0}: Error finding container f0e4d9d6ae4dfcc0dc4af0b30f265c090dbeacbf2208bc70dcfe8240828e80d3: Status 404 returned error can't find the container with id f0e4d9d6ae4dfcc0dc4af0b30f265c090dbeacbf2208bc70dcfe8240828e80d3
Jul 29 04:26:44.580667 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:26:44.580644    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dd29cdf8-59b7ddcf44-9lsnc"]
Jul 29 04:26:44.592915 ip-10-0-135-138 kubenswrapper[2570]: W0729 04:26:44.592890    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21a6f7e6_e3d7_410b_a737_f223d4f9d76b.slice/crio-019ca1be68851b387698eae55d5e3424e6f8d407f4cfef5667b885fdfd3e5024 WatchSource:0}: Error finding container 019ca1be68851b387698eae55d5e3424e6f8d407f4cfef5667b885fdfd3e5024: Status 404 returned error can't find the container with id 019ca1be68851b387698eae55d5e3424e6f8d407f4cfef5667b885fdfd3e5024
Jul 29 04:26:44.805300 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:26:44.805265    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-dd29cdf8-5b4d48784-rb9rv" event={"ID":"325af13c-07c6-47b6-8176-ef1567ed68b7","Type":"ContainerStarted","Data":"eb5399d3926103cba89ab2413fb1dd2ab6ea19756a3edc6b468506973a230591"}
Jul 29 04:26:44.805643 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:26:44.805320    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-dd29cdf8-5b4d48784-rb9rv" event={"ID":"325af13c-07c6-47b6-8176-ef1567ed68b7","Type":"ContainerStarted","Data":"f0e4d9d6ae4dfcc0dc4af0b30f265c090dbeacbf2208bc70dcfe8240828e80d3"}
Jul 29 04:26:44.805643 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:26:44.805448    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-dd29cdf8-5b4d48784-rb9rv"
Jul 29 04:26:44.806984 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:26:44.806954    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-dd29cdf8-59b7ddcf44-9lsnc" event={"ID":"21a6f7e6-e3d7-410b-a737-f223d4f9d76b","Type":"ContainerStarted","Data":"a589dfb7e80568169464acda9a11d3428eb4f790d351bb0ab3933b1d9c4c5e57"}
Jul 29 04:26:44.807129 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:26:44.807009    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-dd29cdf8-59b7ddcf44-9lsnc"
Jul 29 04:26:44.807129 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:26:44.807028    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-dd29cdf8-59b7ddcf44-9lsnc" event={"ID":"21a6f7e6-e3d7-410b-a737-f223d4f9d76b","Type":"ContainerStarted","Data":"019ca1be68851b387698eae55d5e3424e6f8d407f4cfef5667b885fdfd3e5024"}
Jul 29 04:26:44.820671 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:26:44.820627    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-dd29cdf8-5b4d48784-rb9rv" podStartSLOduration=0.820612273 podStartE2EDuration="820.612273ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:26:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:26:44.820070679 +0000 UTC m=+1869.550201772" watchObservedRunningTime="2026-07-29 04:26:44.820612273 +0000 UTC m=+1869.550743373"
Jul 29 04:26:44.836017 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:26:44.835960    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-dd29cdf8-59b7ddcf44-9lsnc" podStartSLOduration=0.835945933 podStartE2EDuration="835.945933ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:26:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:26:44.834699948 +0000 UTC m=+1869.564831032" watchObservedRunningTime="2026-07-29 04:26:44.835945933 +0000 UTC m=+1869.566077002"
Jul 29 04:26:55.813884 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:26:55.813846    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-dd29cdf8-59b7ddcf44-9lsnc"
Jul 29 04:26:55.816179 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:26:55.813931    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-dd29cdf8-5b4d48784-rb9rv"
Jul 29 04:27:08.824195 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:08.824103    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dd29cdf8-b2wzt"]
Jul 29 04:27:08.827734 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:08.827711    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dd29cdf8-b2wzt"
Jul 29 04:27:08.830374 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:08.830352    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 29 04:27:08.830487 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:08.830468    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8frm\""
Jul 29 04:27:08.834375 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:08.834352    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dd29cdf8-b2wzt"]
Jul 29 04:27:08.871316 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:08.871286    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6eaa061-a9b3-4940-a453-4811dbd1648e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-dd29cdf8-b2wzt\" (UID: \"c6eaa061-a9b3-4940-a453-4811dbd1648e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dd29cdf8-b2wzt"
Jul 29 04:27:08.972778 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:08.972734    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6eaa061-a9b3-4940-a453-4811dbd1648e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-dd29cdf8-b2wzt\" (UID: \"c6eaa061-a9b3-4940-a453-4811dbd1648e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dd29cdf8-b2wzt"
Jul 29 04:27:08.973384 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:08.973360    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6eaa061-a9b3-4940-a453-4811dbd1648e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-dd29cdf8-b2wzt\" (UID: \"c6eaa061-a9b3-4940-a453-4811dbd1648e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dd29cdf8-b2wzt"
Jul 29 04:27:09.140063 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:09.140030    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dd29cdf8-b2wzt"
Jul 29 04:27:09.267395 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:09.267370    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dd29cdf8-b2wzt"]
Jul 29 04:27:09.269542 ip-10-0-135-138 kubenswrapper[2570]: W0729 04:27:09.269505    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6eaa061_a9b3_4940_a453_4811dbd1648e.slice/crio-fd2e2f1724419409c49cb5c7f4f55c0768e4167e94c3e954490d93f9bddf60bf WatchSource:0}: Error finding container fd2e2f1724419409c49cb5c7f4f55c0768e4167e94c3e954490d93f9bddf60bf: Status 404 returned error can't find the container with id fd2e2f1724419409c49cb5c7f4f55c0768e4167e94c3e954490d93f9bddf60bf
Jul 29 04:27:09.274179 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:09.274157    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 04:27:09.928672 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:09.928632    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dd29cdf8-b2wzt" event={"ID":"c6eaa061-a9b3-4940-a453-4811dbd1648e","Type":"ContainerStarted","Data":"fd2e2f1724419409c49cb5c7f4f55c0768e4167e94c3e954490d93f9bddf60bf"}
Jul 29 04:27:11.940693 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:11.940651    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dd29cdf8-b2wzt" event={"ID":"c6eaa061-a9b3-4940-a453-4811dbd1648e","Type":"ContainerStarted","Data":"5fa9dc31a7ffaebefbe74f3f871656fc3041a599595524e9121962c85c28d1f0"}
Jul 29 04:27:11.956377 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:11.956327    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dd29cdf8-b2wzt" podStartSLOduration=1.805057368 podStartE2EDuration="3.956313252s" totalImagesPullingTime="2.151255884s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:27:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:27:11.953988158 +0000 UTC m=+1896.684119251" watchObservedRunningTime="2026-07-29 04:27:11.956313252 +0000 UTC m=+1896.686444322"
Jul 29 04:27:12.947829 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:12.947777    2570 generic.go:356] "Generic (PLEG): container finished" podID="c6eaa061-a9b3-4940-a453-4811dbd1648e" containerID="5fa9dc31a7ffaebefbe74f3f871656fc3041a599595524e9121962c85c28d1f0" exitCode=0
Jul 29 04:27:12.948302 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:12.947870    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dd29cdf8-b2wzt" event={"ID":"c6eaa061-a9b3-4940-a453-4811dbd1648e","Type":"ContainerDied","Data":"5fa9dc31a7ffaebefbe74f3f871656fc3041a599595524e9121962c85c28d1f0"}
Jul 29 04:27:14.078218 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:14.078190    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dd29cdf8-b2wzt"
Jul 29 04:27:14.120409 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:14.120385    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6eaa061-a9b3-4940-a453-4811dbd1648e-maas-api-service-ca\") pod \"c6eaa061-a9b3-4940-a453-4811dbd1648e\" (UID: \"c6eaa061-a9b3-4940-a453-4811dbd1648e\") "
Jul 29 04:27:14.120770 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:14.120743    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6eaa061-a9b3-4940-a453-4811dbd1648e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c6eaa061-a9b3-4940-a453-4811dbd1648e" (UID: "c6eaa061-a9b3-4940-a453-4811dbd1648e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 04:27:14.222058 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:14.221988    2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6eaa061-a9b3-4940-a453-4811dbd1648e-maas-api-service-ca\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:27:14.957950 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:14.957910    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dd29cdf8-b2wzt" event={"ID":"c6eaa061-a9b3-4940-a453-4811dbd1648e","Type":"ContainerDied","Data":"fd2e2f1724419409c49cb5c7f4f55c0768e4167e94c3e954490d93f9bddf60bf"}
Jul 29 04:27:14.957950 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:14.957956    2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd2e2f1724419409c49cb5c7f4f55c0768e4167e94c3e954490d93f9bddf60bf"
Jul 29 04:27:14.958159 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:14.957918    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dd29cdf8-b2wzt"
Jul 29 04:27:23.961351 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:23.961316    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dd29cdf8-59b7ddcf44-9lsnc"]
Jul 29 04:27:23.961924 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:23.961566    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-dd29cdf8-59b7ddcf44-9lsnc" podUID="21a6f7e6-e3d7-410b-a737-f223d4f9d76b" containerName="payload-processing" containerID="cri-o://a589dfb7e80568169464acda9a11d3428eb4f790d351bb0ab3933b1d9c4c5e57" gracePeriod=30
Jul 29 04:27:23.970666 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:23.970636    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dd29cdf8-5b4d48784-rb9rv"]
Jul 29 04:27:23.970918 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:23.970889    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-dd29cdf8-5b4d48784-rb9rv" podUID="325af13c-07c6-47b6-8176-ef1567ed68b7" containerName="payload-pre-processing" containerID="cri-o://eb5399d3926103cba89ab2413fb1dd2ab6ea19756a3edc6b468506973a230591" gracePeriod=30
Jul 29 04:27:24.086106 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:24.086072    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dd29cdf8-b2wzt"]
Jul 29 04:27:24.088280 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:24.088257    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dd29cdf8-b2wzt"]
Jul 29 04:27:24.259169 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:24.259146    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-dd29cdf8-59b7ddcf44-9lsnc"
Jul 29 04:27:24.262024 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:24.262008    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-dd29cdf8-5b4d48784-rb9rv"
Jul 29 04:27:24.413522 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:24.413497    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vjjch\" (UniqueName: \"kubernetes.io/projected/21a6f7e6-e3d7-410b-a737-f223d4f9d76b-kube-api-access-vjjch\") pod \"21a6f7e6-e3d7-410b-a737-f223d4f9d76b\" (UID: \"21a6f7e6-e3d7-410b-a737-f223d4f9d76b\") "
Jul 29 04:27:24.413693 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:24.413571    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21a6f7e6-e3d7-410b-a737-f223d4f9d76b-plugins-config-volume\") pod \"21a6f7e6-e3d7-410b-a737-f223d4f9d76b\" (UID: \"21a6f7e6-e3d7-410b-a737-f223d4f9d76b\") "
Jul 29 04:27:24.413693 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:24.413626    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/325af13c-07c6-47b6-8176-ef1567ed68b7-plugins-config-volume\") pod \"325af13c-07c6-47b6-8176-ef1567ed68b7\" (UID: \"325af13c-07c6-47b6-8176-ef1567ed68b7\") "
Jul 29 04:27:24.413693 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:24.413648    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-76fn8\" (UniqueName: \"kubernetes.io/projected/325af13c-07c6-47b6-8176-ef1567ed68b7-kube-api-access-76fn8\") pod \"325af13c-07c6-47b6-8176-ef1567ed68b7\" (UID: \"325af13c-07c6-47b6-8176-ef1567ed68b7\") "
Jul 29 04:27:24.414004 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:24.413976    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21a6f7e6-e3d7-410b-a737-f223d4f9d76b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "21a6f7e6-e3d7-410b-a737-f223d4f9d76b" (UID: "21a6f7e6-e3d7-410b-a737-f223d4f9d76b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 04:27:24.414091 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:24.414040    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/325af13c-07c6-47b6-8176-ef1567ed68b7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "325af13c-07c6-47b6-8176-ef1567ed68b7" (UID: "325af13c-07c6-47b6-8176-ef1567ed68b7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 04:27:24.416022 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:24.415989    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/325af13c-07c6-47b6-8176-ef1567ed68b7-kube-api-access-76fn8" (OuterVolumeSpecName: "kube-api-access-76fn8") pod "325af13c-07c6-47b6-8176-ef1567ed68b7" (UID: "325af13c-07c6-47b6-8176-ef1567ed68b7"). InnerVolumeSpecName "kube-api-access-76fn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 04:27:24.416022 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:24.415992    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21a6f7e6-e3d7-410b-a737-f223d4f9d76b-kube-api-access-vjjch" (OuterVolumeSpecName: "kube-api-access-vjjch") pod "21a6f7e6-e3d7-410b-a737-f223d4f9d76b" (UID: "21a6f7e6-e3d7-410b-a737-f223d4f9d76b"). InnerVolumeSpecName "kube-api-access-vjjch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 04:27:24.515016 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:24.514934    2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21a6f7e6-e3d7-410b-a737-f223d4f9d76b-plugins-config-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:27:24.515016 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:24.514961    2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/325af13c-07c6-47b6-8176-ef1567ed68b7-plugins-config-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:27:24.515016 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:24.514970    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76fn8\" (UniqueName: \"kubernetes.io/projected/325af13c-07c6-47b6-8176-ef1567ed68b7-kube-api-access-76fn8\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:27:24.515016 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:24.514979    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjjch\" (UniqueName: \"kubernetes.io/projected/21a6f7e6-e3d7-410b-a737-f223d4f9d76b-kube-api-access-vjjch\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:27:25.016627 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:25.016590    2570 generic.go:356] "Generic (PLEG): container finished" podID="21a6f7e6-e3d7-410b-a737-f223d4f9d76b" containerID="a589dfb7e80568169464acda9a11d3428eb4f790d351bb0ab3933b1d9c4c5e57" exitCode=0
Jul 29 04:27:25.017101 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:25.016631    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-dd29cdf8-59b7ddcf44-9lsnc"
Jul 29 04:27:25.017101 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:25.016672    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-dd29cdf8-59b7ddcf44-9lsnc" event={"ID":"21a6f7e6-e3d7-410b-a737-f223d4f9d76b","Type":"ContainerDied","Data":"a589dfb7e80568169464acda9a11d3428eb4f790d351bb0ab3933b1d9c4c5e57"}
Jul 29 04:27:25.017101 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:25.016732    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-dd29cdf8-59b7ddcf44-9lsnc" event={"ID":"21a6f7e6-e3d7-410b-a737-f223d4f9d76b","Type":"ContainerDied","Data":"019ca1be68851b387698eae55d5e3424e6f8d407f4cfef5667b885fdfd3e5024"}
Jul 29 04:27:25.017101 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:25.016767    2570 scope.go:117] "RemoveContainer" containerID="a589dfb7e80568169464acda9a11d3428eb4f790d351bb0ab3933b1d9c4c5e57"
Jul 29 04:27:25.018411 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:25.018388    2570 generic.go:356] "Generic (PLEG): container finished" podID="325af13c-07c6-47b6-8176-ef1567ed68b7" containerID="eb5399d3926103cba89ab2413fb1dd2ab6ea19756a3edc6b468506973a230591" exitCode=0
Jul 29 04:27:25.018680 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:25.018440    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-dd29cdf8-5b4d48784-rb9rv"
Jul 29 04:27:25.018680 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:25.018477    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-dd29cdf8-5b4d48784-rb9rv" event={"ID":"325af13c-07c6-47b6-8176-ef1567ed68b7","Type":"ContainerDied","Data":"eb5399d3926103cba89ab2413fb1dd2ab6ea19756a3edc6b468506973a230591"}
Jul 29 04:27:25.018680 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:25.018516    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-dd29cdf8-5b4d48784-rb9rv" event={"ID":"325af13c-07c6-47b6-8176-ef1567ed68b7","Type":"ContainerDied","Data":"f0e4d9d6ae4dfcc0dc4af0b30f265c090dbeacbf2208bc70dcfe8240828e80d3"}
Jul 29 04:27:25.026867 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:25.026851    2570 scope.go:117] "RemoveContainer" containerID="a589dfb7e80568169464acda9a11d3428eb4f790d351bb0ab3933b1d9c4c5e57"
Jul 29 04:27:25.027157 ip-10-0-135-138 kubenswrapper[2570]: E0729 04:27:25.027126    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a589dfb7e80568169464acda9a11d3428eb4f790d351bb0ab3933b1d9c4c5e57\": container with ID starting with a589dfb7e80568169464acda9a11d3428eb4f790d351bb0ab3933b1d9c4c5e57 not found: ID does not exist" containerID="a589dfb7e80568169464acda9a11d3428eb4f790d351bb0ab3933b1d9c4c5e57"
Jul 29 04:27:25.027290 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:25.027159    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a589dfb7e80568169464acda9a11d3428eb4f790d351bb0ab3933b1d9c4c5e57"} err="failed to get container status \"a589dfb7e80568169464acda9a11d3428eb4f790d351bb0ab3933b1d9c4c5e57\": rpc error: code = NotFound desc = could not find container \"a589dfb7e80568169464acda9a11d3428eb4f790d351bb0ab3933b1d9c4c5e57\": container with ID starting with a589dfb7e80568169464acda9a11d3428eb4f790d351bb0ab3933b1d9c4c5e57 not found: ID does not exist"
Jul 29 04:27:25.027290 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:25.027181    2570 scope.go:117] "RemoveContainer" containerID="eb5399d3926103cba89ab2413fb1dd2ab6ea19756a3edc6b468506973a230591"
Jul 29 04:27:25.035270 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:25.035255    2570 scope.go:117] "RemoveContainer" containerID="eb5399d3926103cba89ab2413fb1dd2ab6ea19756a3edc6b468506973a230591"
Jul 29 04:27:25.035506 ip-10-0-135-138 kubenswrapper[2570]: E0729 04:27:25.035487    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb5399d3926103cba89ab2413fb1dd2ab6ea19756a3edc6b468506973a230591\": container with ID starting with eb5399d3926103cba89ab2413fb1dd2ab6ea19756a3edc6b468506973a230591 not found: ID does not exist" containerID="eb5399d3926103cba89ab2413fb1dd2ab6ea19756a3edc6b468506973a230591"
Jul 29 04:27:25.035561 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:25.035512    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb5399d3926103cba89ab2413fb1dd2ab6ea19756a3edc6b468506973a230591"} err="failed to get container status \"eb5399d3926103cba89ab2413fb1dd2ab6ea19756a3edc6b468506973a230591\": rpc error: code = NotFound desc = could not find container \"eb5399d3926103cba89ab2413fb1dd2ab6ea19756a3edc6b468506973a230591\": container with ID starting with eb5399d3926103cba89ab2413fb1dd2ab6ea19756a3edc6b468506973a230591 not found: ID does not exist"
Jul 29 04:27:25.041085 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:25.041065    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dd29cdf8-59b7ddcf44-9lsnc"]
Jul 29 04:27:25.044345 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:25.044325    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dd29cdf8-59b7ddcf44-9lsnc"]
Jul 29 04:27:25.054682 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:25.054662    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dd29cdf8-5b4d48784-rb9rv"]
Jul 29 04:27:25.061597 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:25.061577    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dd29cdf8-5b4d48784-rb9rv"]
Jul 29 04:27:25.836308 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:25.836276    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21a6f7e6-e3d7-410b-a737-f223d4f9d76b" path="/var/lib/kubelet/pods/21a6f7e6-e3d7-410b-a737-f223d4f9d76b/volumes"
Jul 29 04:27:25.836636 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:25.836623    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="325af13c-07c6-47b6-8176-ef1567ed68b7" path="/var/lib/kubelet/pods/325af13c-07c6-47b6-8176-ef1567ed68b7/volumes"
Jul 29 04:27:25.837124 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:25.837110    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6eaa061-a9b3-4940-a453-4811dbd1648e" path="/var/lib/kubelet/pods/c6eaa061-a9b3-4940-a453-4811dbd1648e/volumes"
Jul 29 04:27:27.610090 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:27.610056    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-74690011-74878c7b76-plgbl"]
Jul 29 04:27:27.610547 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:27.610486    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="325af13c-07c6-47b6-8176-ef1567ed68b7" containerName="payload-pre-processing"
Jul 29 04:27:27.610547 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:27.610499    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="325af13c-07c6-47b6-8176-ef1567ed68b7" containerName="payload-pre-processing"
Jul 29 04:27:27.610547 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:27.610510    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6eaa061-a9b3-4940-a453-4811dbd1648e" containerName="revoke-keys"
Jul 29 04:27:27.610547 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:27.610515    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6eaa061-a9b3-4940-a453-4811dbd1648e" containerName="revoke-keys"
Jul 29 04:27:27.610547 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:27.610536    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21a6f7e6-e3d7-410b-a737-f223d4f9d76b" containerName="payload-processing"
Jul 29 04:27:27.610547 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:27.610542    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="21a6f7e6-e3d7-410b-a737-f223d4f9d76b" containerName="payload-processing"
Jul 29 04:27:27.610906 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:27.610616    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="21a6f7e6-e3d7-410b-a737-f223d4f9d76b" containerName="payload-processing"
Jul 29 04:27:27.610906 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:27.610624    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6eaa061-a9b3-4940-a453-4811dbd1648e" containerName="revoke-keys"
Jul 29 04:27:27.610906 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:27.610631    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="325af13c-07c6-47b6-8176-ef1567ed68b7" containerName="payload-pre-processing"
Jul 29 04:27:27.615383 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:27.615359    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-74690011-74878c7b76-plgbl"
Jul 29 04:27:27.618381 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:27.618356    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 29 04:27:27.618521 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:27.618356    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 29 04:27:27.618521 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:27.618362    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-74690011-serving-cert\""
Jul 29 04:27:27.619565 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:27.619548    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-lh6p9\""
Jul 29 04:27:27.623426 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:27.623405    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-74690011-74878c7b76-plgbl"]
Jul 29 04:27:27.747247 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:27.747216    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/765426b8-3781-4988-9732-a041d5940e07-maas-api-tls\") pod \"maas-api-e2e-migrate-74690011-74878c7b76-plgbl\" (UID: \"765426b8-3781-4988-9732-a041d5940e07\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-74690011-74878c7b76-plgbl"
Jul 29 04:27:27.747425 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:27.747258    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6lj6\" (UniqueName: \"kubernetes.io/projected/765426b8-3781-4988-9732-a041d5940e07-kube-api-access-t6lj6\") pod \"maas-api-e2e-migrate-74690011-74878c7b76-plgbl\" (UID: \"765426b8-3781-4988-9732-a041d5940e07\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-74690011-74878c7b76-plgbl"
Jul 29 04:27:27.747425 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:27.747344    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/765426b8-3781-4988-9732-a041d5940e07-openshift-service-ca\") pod \"maas-api-e2e-migrate-74690011-74878c7b76-plgbl\" (UID: \"765426b8-3781-4988-9732-a041d5940e07\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-74690011-74878c7b76-plgbl"
Jul 29 04:27:27.848505 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:27.848476    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/765426b8-3781-4988-9732-a041d5940e07-maas-api-tls\") pod \"maas-api-e2e-migrate-74690011-74878c7b76-plgbl\" (UID: \"765426b8-3781-4988-9732-a041d5940e07\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-74690011-74878c7b76-plgbl"
Jul 29 04:27:27.848684 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:27.848512    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6lj6\" (UniqueName: \"kubernetes.io/projected/765426b8-3781-4988-9732-a041d5940e07-kube-api-access-t6lj6\") pod \"maas-api-e2e-migrate-74690011-74878c7b76-plgbl\" (UID: \"765426b8-3781-4988-9732-a041d5940e07\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-74690011-74878c7b76-plgbl"
Jul 29 04:27:27.848684 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:27.848597    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/765426b8-3781-4988-9732-a041d5940e07-openshift-service-ca\") pod \"maas-api-e2e-migrate-74690011-74878c7b76-plgbl\" (UID: \"765426b8-3781-4988-9732-a041d5940e07\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-74690011-74878c7b76-plgbl"
Jul 29 04:27:27.849290 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:27.849261    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/765426b8-3781-4988-9732-a041d5940e07-openshift-service-ca\") pod \"maas-api-e2e-migrate-74690011-74878c7b76-plgbl\" (UID: \"765426b8-3781-4988-9732-a041d5940e07\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-74690011-74878c7b76-plgbl"
Jul 29 04:27:27.850969 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:27.850948    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/765426b8-3781-4988-9732-a041d5940e07-maas-api-tls\") pod \"maas-api-e2e-migrate-74690011-74878c7b76-plgbl\" (UID: \"765426b8-3781-4988-9732-a041d5940e07\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-74690011-74878c7b76-plgbl"
Jul 29 04:27:27.858435 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:27.858415    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6lj6\" (UniqueName: \"kubernetes.io/projected/765426b8-3781-4988-9732-a041d5940e07-kube-api-access-t6lj6\") pod \"maas-api-e2e-migrate-74690011-74878c7b76-plgbl\" (UID: \"765426b8-3781-4988-9732-a041d5940e07\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-74690011-74878c7b76-plgbl"
Jul 29 04:27:27.928357 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:27.928298    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-74690011-74878c7b76-plgbl"
Jul 29 04:27:28.065030 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:28.065004    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-74690011-74878c7b76-plgbl"]
Jul 29 04:27:28.067130 ip-10-0-135-138 kubenswrapper[2570]: W0729 04:27:28.067102    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod765426b8_3781_4988_9732_a041d5940e07.slice/crio-f4660a6c4cd336ca393cac12dcccdfa78393d0d651f45788d9c5f038e57ccc4d WatchSource:0}: Error finding container f4660a6c4cd336ca393cac12dcccdfa78393d0d651f45788d9c5f038e57ccc4d: Status 404 returned error can't find the container with id f4660a6c4cd336ca393cac12dcccdfa78393d0d651f45788d9c5f038e57ccc4d
Jul 29 04:27:28.238916 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:28.238844    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-74690011-7c7f4bfdbd-5lj2s"]
Jul 29 04:27:28.243521 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:28.243505    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-74690011-7c7f4bfdbd-5lj2s"
Jul 29 04:27:28.246609 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:28.246579    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-74690011\""
Jul 29 04:27:28.246609 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:28.246608    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-74690011-dockercfg-6fhh4\""
Jul 29 04:27:28.258886 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:28.258860    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-74690011-7c7f4bfdbd-5lj2s"]
Jul 29 04:27:28.263198 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:28.263160    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-74690011-69f764786f-xzkxx"]
Jul 29 04:27:28.268383 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:28.268324    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-74690011-69f764786f-xzkxx"
Jul 29 04:27:28.275052 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:28.275031    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-74690011-69f764786f-xzkxx"]
Jul 29 04:27:28.353763 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:28.353728    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4314cef7-c274-488f-8609-f13e49d9b8b8-plugins-config-volume\") pod \"payload-processing-e2e-migrate-74690011-69f764786f-xzkxx\" (UID: \"4314cef7-c274-488f-8609-f13e49d9b8b8\") " pod="openshift-ingress/payload-processing-e2e-migrate-74690011-69f764786f-xzkxx"
Jul 29 04:27:28.353964 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:28.353781    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fada5537-42c7-491e-880b-856188c1ab8e-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-74690011-7c7f4bfdbd-5lj2s\" (UID: \"fada5537-42c7-491e-880b-856188c1ab8e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-74690011-7c7f4bfdbd-5lj2s"
Jul 29 04:27:28.353964 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:28.353924    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gbzd\" (UniqueName: \"kubernetes.io/projected/fada5537-42c7-491e-880b-856188c1ab8e-kube-api-access-4gbzd\") pod \"payload-pre-processing-e2e-migrate-74690011-7c7f4bfdbd-5lj2s\" (UID: \"fada5537-42c7-491e-880b-856188c1ab8e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-74690011-7c7f4bfdbd-5lj2s"
Jul 29 04:27:28.353964 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:28.353958    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgjvg\" (UniqueName: \"kubernetes.io/projected/4314cef7-c274-488f-8609-f13e49d9b8b8-kube-api-access-xgjvg\") pod \"payload-processing-e2e-migrate-74690011-69f764786f-xzkxx\" (UID: \"4314cef7-c274-488f-8609-f13e49d9b8b8\") " pod="openshift-ingress/payload-processing-e2e-migrate-74690011-69f764786f-xzkxx"
Jul 29 04:27:28.454565 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:28.454533    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4314cef7-c274-488f-8609-f13e49d9b8b8-plugins-config-volume\") pod \"payload-processing-e2e-migrate-74690011-69f764786f-xzkxx\" (UID: \"4314cef7-c274-488f-8609-f13e49d9b8b8\") " pod="openshift-ingress/payload-processing-e2e-migrate-74690011-69f764786f-xzkxx"
Jul 29 04:27:28.454737 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:28.454576    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fada5537-42c7-491e-880b-856188c1ab8e-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-74690011-7c7f4bfdbd-5lj2s\" (UID: \"fada5537-42c7-491e-880b-856188c1ab8e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-74690011-7c7f4bfdbd-5lj2s"
Jul 29 04:27:28.454737 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:28.454652    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gbzd\" (UniqueName: \"kubernetes.io/projected/fada5537-42c7-491e-880b-856188c1ab8e-kube-api-access-4gbzd\") pod \"payload-pre-processing-e2e-migrate-74690011-7c7f4bfdbd-5lj2s\" (UID: \"fada5537-42c7-491e-880b-856188c1ab8e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-74690011-7c7f4bfdbd-5lj2s"
Jul 29 04:27:28.454737 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:28.454683    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgjvg\" (UniqueName: \"kubernetes.io/projected/4314cef7-c274-488f-8609-f13e49d9b8b8-kube-api-access-xgjvg\") pod \"payload-processing-e2e-migrate-74690011-69f764786f-xzkxx\" (UID: \"4314cef7-c274-488f-8609-f13e49d9b8b8\") " pod="openshift-ingress/payload-processing-e2e-migrate-74690011-69f764786f-xzkxx"
Jul 29 04:27:28.455369 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:28.455348    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fada5537-42c7-491e-880b-856188c1ab8e-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-74690011-7c7f4bfdbd-5lj2s\" (UID: \"fada5537-42c7-491e-880b-856188c1ab8e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-74690011-7c7f4bfdbd-5lj2s"
Jul 29 04:27:28.455411 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:28.455373    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4314cef7-c274-488f-8609-f13e49d9b8b8-plugins-config-volume\") pod \"payload-processing-e2e-migrate-74690011-69f764786f-xzkxx\" (UID: \"4314cef7-c274-488f-8609-f13e49d9b8b8\") " pod="openshift-ingress/payload-processing-e2e-migrate-74690011-69f764786f-xzkxx"
Jul 29 04:27:28.463031 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:28.463006    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgjvg\" (UniqueName: \"kubernetes.io/projected/4314cef7-c274-488f-8609-f13e49d9b8b8-kube-api-access-xgjvg\") pod \"payload-processing-e2e-migrate-74690011-69f764786f-xzkxx\" (UID: \"4314cef7-c274-488f-8609-f13e49d9b8b8\") " pod="openshift-ingress/payload-processing-e2e-migrate-74690011-69f764786f-xzkxx"
Jul 29 04:27:28.463146 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:28.463006    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gbzd\" (UniqueName: \"kubernetes.io/projected/fada5537-42c7-491e-880b-856188c1ab8e-kube-api-access-4gbzd\") pod \"payload-pre-processing-e2e-migrate-74690011-7c7f4bfdbd-5lj2s\" (UID: \"fada5537-42c7-491e-880b-856188c1ab8e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-74690011-7c7f4bfdbd-5lj2s"
Jul 29 04:27:28.554388 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:28.554233    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-74690011-7c7f4bfdbd-5lj2s"
Jul 29 04:27:28.583523 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:28.582754    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-74690011-69f764786f-xzkxx"
Jul 29 04:27:28.738598 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:28.738521    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-74690011-7c7f4bfdbd-5lj2s"]
Jul 29 04:27:28.745351 ip-10-0-135-138 kubenswrapper[2570]: W0729 04:27:28.745318    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfada5537_42c7_491e_880b_856188c1ab8e.slice/crio-c25948984a14b75e29c47fe0682562182742344e8bffbf4bc4b05a45de455131 WatchSource:0}: Error finding container c25948984a14b75e29c47fe0682562182742344e8bffbf4bc4b05a45de455131: Status 404 returned error can't find the container with id c25948984a14b75e29c47fe0682562182742344e8bffbf4bc4b05a45de455131
Jul 29 04:27:28.774314 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:28.774282    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-74690011-69f764786f-xzkxx"]
Jul 29 04:27:28.776220 ip-10-0-135-138 kubenswrapper[2570]: W0729 04:27:28.776182    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4314cef7_c274_488f_8609_f13e49d9b8b8.slice/crio-354b053d28f00684ba041b8d5c41cc20ad4bb57c38cc041d80fc9ca6cf31fe26 WatchSource:0}: Error finding container 354b053d28f00684ba041b8d5c41cc20ad4bb57c38cc041d80fc9ca6cf31fe26: Status 404 returned error can't find the container with id 354b053d28f00684ba041b8d5c41cc20ad4bb57c38cc041d80fc9ca6cf31fe26
Jul 29 04:27:29.046448 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:29.046384    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-74690011-7c7f4bfdbd-5lj2s" event={"ID":"fada5537-42c7-491e-880b-856188c1ab8e","Type":"ContainerStarted","Data":"976be568438cc6e473546e718e8b39f01c47380455b38c2d3ed8a42274618473"}
Jul 29 04:27:29.046448 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:29.046449    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-74690011-7c7f4bfdbd-5lj2s" event={"ID":"fada5537-42c7-491e-880b-856188c1ab8e","Type":"ContainerStarted","Data":"c25948984a14b75e29c47fe0682562182742344e8bffbf4bc4b05a45de455131"}
Jul 29 04:27:29.046749 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:29.046481    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-74690011-7c7f4bfdbd-5lj2s"
Jul 29 04:27:29.048211 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:29.048183    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-74690011-69f764786f-xzkxx" event={"ID":"4314cef7-c274-488f-8609-f13e49d9b8b8","Type":"ContainerStarted","Data":"ceb505cb44ba54b6cc4eee26ecb40dc6724fc446370d422c1c7cef2b850917ad"}
Jul 29 04:27:29.048578 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:29.048235    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-74690011-69f764786f-xzkxx" event={"ID":"4314cef7-c274-488f-8609-f13e49d9b8b8","Type":"ContainerStarted","Data":"354b053d28f00684ba041b8d5c41cc20ad4bb57c38cc041d80fc9ca6cf31fe26"}
Jul 29 04:27:29.048578 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:29.048439    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-74690011-69f764786f-xzkxx"
Jul 29 04:27:29.049768 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:29.049707    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-74690011-74878c7b76-plgbl" event={"ID":"765426b8-3781-4988-9732-a041d5940e07","Type":"ContainerStarted","Data":"f4660a6c4cd336ca393cac12dcccdfa78393d0d651f45788d9c5f038e57ccc4d"}
Jul 29 04:27:29.080851 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:29.080784    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-74690011-7c7f4bfdbd-5lj2s" podStartSLOduration=1.080765271 podStartE2EDuration="1.080765271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:27:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:27:29.062040217 +0000 UTC m=+1913.792171288" watchObservedRunningTime="2026-07-29 04:27:29.080765271 +0000 UTC m=+1913.810896341"
Jul 29 04:27:29.080993 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:29.080909    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-74690011-69f764786f-xzkxx" podStartSLOduration=1.080899329 podStartE2EDuration="1.080899329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:27:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:27:29.079282175 +0000 UTC m=+1913.809413270" watchObservedRunningTime="2026-07-29 04:27:29.080899329 +0000 UTC m=+1913.811030399"
Jul 29 04:27:31.065048 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:31.065002    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-74690011-74878c7b76-plgbl" event={"ID":"765426b8-3781-4988-9732-a041d5940e07","Type":"ContainerStarted","Data":"e8dc98de22434e4dcd62982ac523576d69a5cd877f0c7291327f31241f958336"}
Jul 29 04:27:31.065430 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:31.065067    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-74690011-74878c7b76-plgbl"
Jul 29 04:27:31.082055 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:31.081992    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-74690011-74878c7b76-plgbl" podStartSLOduration=1.824451555 podStartE2EDuration="4.081951791s" totalImagesPullingTime="2.257500236s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:27:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:27:31.081118265 +0000 UTC m=+1915.811249395" watchObservedRunningTime="2026-07-29 04:27:31.081951791 +0000 UTC m=+1915.812082862"
Jul 29 04:27:37.075409 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:37.075380    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-74690011-74878c7b76-plgbl"
Jul 29 04:27:40.058200 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:40.058171    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-74690011-69f764786f-xzkxx"
Jul 29 04:27:40.058639 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:40.058341    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-74690011-7c7f4bfdbd-5lj2s"
Jul 29 04:27:59.020407 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:59.020256    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-74690011-74878c7b76-plgbl"]
Jul 29 04:27:59.020955 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:59.020540    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-74690011-74878c7b76-plgbl" podUID="765426b8-3781-4988-9732-a041d5940e07" containerName="maas-api" containerID="cri-o://e8dc98de22434e4dcd62982ac523576d69a5cd877f0c7291327f31241f958336" gracePeriod=30
Jul 29 04:27:59.066410 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:59.066383    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-74690011-69f764786f-xzkxx"]
Jul 29 04:27:59.066595 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:59.066574    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-74690011-69f764786f-xzkxx" podUID="4314cef7-c274-488f-8609-f13e49d9b8b8" containerName="payload-processing" containerID="cri-o://ceb505cb44ba54b6cc4eee26ecb40dc6724fc446370d422c1c7cef2b850917ad" gracePeriod=30
Jul 29 04:27:59.075045 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:59.075006    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-74690011-7c7f4bfdbd-5lj2s"]
Jul 29 04:27:59.075320 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:59.075280    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-74690011-7c7f4bfdbd-5lj2s" podUID="fada5537-42c7-491e-880b-856188c1ab8e" containerName="payload-pre-processing" containerID="cri-o://976be568438cc6e473546e718e8b39f01c47380455b38c2d3ed8a42274618473" gracePeriod=30
Jul 29 04:27:59.215324 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:59.215253    2570 generic.go:356] "Generic (PLEG): container finished" podID="765426b8-3781-4988-9732-a041d5940e07" containerID="e8dc98de22434e4dcd62982ac523576d69a5cd877f0c7291327f31241f958336" exitCode=0
Jul 29 04:27:59.215543 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:59.215488    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-74690011-74878c7b76-plgbl" event={"ID":"765426b8-3781-4988-9732-a041d5940e07","Type":"ContainerDied","Data":"e8dc98de22434e4dcd62982ac523576d69a5cd877f0c7291327f31241f958336"}
Jul 29 04:27:59.218608 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:59.218534    2570 generic.go:356] "Generic (PLEG): container finished" podID="fada5537-42c7-491e-880b-856188c1ab8e" containerID="976be568438cc6e473546e718e8b39f01c47380455b38c2d3ed8a42274618473" exitCode=0
Jul 29 04:27:59.218737 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:59.218623    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-74690011-7c7f4bfdbd-5lj2s" event={"ID":"fada5537-42c7-491e-880b-856188c1ab8e","Type":"ContainerDied","Data":"976be568438cc6e473546e718e8b39f01c47380455b38c2d3ed8a42274618473"}
Jul 29 04:27:59.220599 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:59.220574    2570 generic.go:356] "Generic (PLEG): container finished" podID="4314cef7-c274-488f-8609-f13e49d9b8b8" containerID="ceb505cb44ba54b6cc4eee26ecb40dc6724fc446370d422c1c7cef2b850917ad" exitCode=0
Jul 29 04:27:59.220713 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:59.220617    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-74690011-69f764786f-xzkxx" event={"ID":"4314cef7-c274-488f-8609-f13e49d9b8b8","Type":"ContainerDied","Data":"ceb505cb44ba54b6cc4eee26ecb40dc6724fc446370d422c1c7cef2b850917ad"}
Jul 29 04:27:59.352326 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:59.352294    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-74690011-69f764786f-xzkxx"
Jul 29 04:27:59.379126 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:59.379103    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-74690011-7c7f4bfdbd-5lj2s"
Jul 29 04:27:59.382314 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:59.382294    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-74690011-74878c7b76-plgbl"
Jul 29 04:27:59.446555 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:59.446522    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4gbzd\" (UniqueName: \"kubernetes.io/projected/fada5537-42c7-491e-880b-856188c1ab8e-kube-api-access-4gbzd\") pod \"fada5537-42c7-491e-880b-856188c1ab8e\" (UID: \"fada5537-42c7-491e-880b-856188c1ab8e\") "
Jul 29 04:27:59.446735 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:59.446597    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/765426b8-3781-4988-9732-a041d5940e07-maas-api-tls\") pod \"765426b8-3781-4988-9732-a041d5940e07\" (UID: \"765426b8-3781-4988-9732-a041d5940e07\") "
Jul 29 04:27:59.446735 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:59.446618    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t6lj6\" (UniqueName: \"kubernetes.io/projected/765426b8-3781-4988-9732-a041d5940e07-kube-api-access-t6lj6\") pod \"765426b8-3781-4988-9732-a041d5940e07\" (UID: \"765426b8-3781-4988-9732-a041d5940e07\") "
Jul 29 04:27:59.446735 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:59.446635    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/765426b8-3781-4988-9732-a041d5940e07-openshift-service-ca\") pod \"765426b8-3781-4988-9732-a041d5940e07\" (UID: \"765426b8-3781-4988-9732-a041d5940e07\") "
Jul 29 04:27:59.446735 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:59.446653    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xgjvg\" (UniqueName: \"kubernetes.io/projected/4314cef7-c274-488f-8609-f13e49d9b8b8-kube-api-access-xgjvg\") pod \"4314cef7-c274-488f-8609-f13e49d9b8b8\" (UID: \"4314cef7-c274-488f-8609-f13e49d9b8b8\") "
Jul 29 04:27:59.446735 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:59.446684    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4314cef7-c274-488f-8609-f13e49d9b8b8-plugins-config-volume\") pod \"4314cef7-c274-488f-8609-f13e49d9b8b8\" (UID: \"4314cef7-c274-488f-8609-f13e49d9b8b8\") "
Jul 29 04:27:59.446735 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:59.446733    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fada5537-42c7-491e-880b-856188c1ab8e-plugins-config-volume\") pod \"fada5537-42c7-491e-880b-856188c1ab8e\" (UID: \"fada5537-42c7-491e-880b-856188c1ab8e\") "
Jul 29 04:27:59.447286 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:59.447227    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fada5537-42c7-491e-880b-856188c1ab8e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fada5537-42c7-491e-880b-856188c1ab8e" (UID: "fada5537-42c7-491e-880b-856188c1ab8e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 04:27:59.447286 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:59.447240    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/765426b8-3781-4988-9732-a041d5940e07-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "765426b8-3781-4988-9732-a041d5940e07" (UID: "765426b8-3781-4988-9732-a041d5940e07"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 04:27:59.447286 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:59.447266    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4314cef7-c274-488f-8609-f13e49d9b8b8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4314cef7-c274-488f-8609-f13e49d9b8b8" (UID: "4314cef7-c274-488f-8609-f13e49d9b8b8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 04:27:59.449069 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:59.449036    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4314cef7-c274-488f-8609-f13e49d9b8b8-kube-api-access-xgjvg" (OuterVolumeSpecName: "kube-api-access-xgjvg") pod "4314cef7-c274-488f-8609-f13e49d9b8b8" (UID: "4314cef7-c274-488f-8609-f13e49d9b8b8"). InnerVolumeSpecName "kube-api-access-xgjvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 04:27:59.449173 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:59.449082    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fada5537-42c7-491e-880b-856188c1ab8e-kube-api-access-4gbzd" (OuterVolumeSpecName: "kube-api-access-4gbzd") pod "fada5537-42c7-491e-880b-856188c1ab8e" (UID: "fada5537-42c7-491e-880b-856188c1ab8e"). InnerVolumeSpecName "kube-api-access-4gbzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 04:27:59.449173 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:59.449099    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/765426b8-3781-4988-9732-a041d5940e07-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "765426b8-3781-4988-9732-a041d5940e07" (UID: "765426b8-3781-4988-9732-a041d5940e07"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 04:27:59.449397 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:59.449383    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/765426b8-3781-4988-9732-a041d5940e07-kube-api-access-t6lj6" (OuterVolumeSpecName: "kube-api-access-t6lj6") pod "765426b8-3781-4988-9732-a041d5940e07" (UID: "765426b8-3781-4988-9732-a041d5940e07"). InnerVolumeSpecName "kube-api-access-t6lj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 04:27:59.547926 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:59.547854    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gbzd\" (UniqueName: \"kubernetes.io/projected/fada5537-42c7-491e-880b-856188c1ab8e-kube-api-access-4gbzd\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:27:59.547926 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:59.547879    2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/765426b8-3781-4988-9732-a041d5940e07-maas-api-tls\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:27:59.547926 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:59.547889    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6lj6\" (UniqueName: \"kubernetes.io/projected/765426b8-3781-4988-9732-a041d5940e07-kube-api-access-t6lj6\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:27:59.547926 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:59.547898    2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/765426b8-3781-4988-9732-a041d5940e07-openshift-service-ca\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:27:59.547926 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:59.547906    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgjvg\" (UniqueName: \"kubernetes.io/projected/4314cef7-c274-488f-8609-f13e49d9b8b8-kube-api-access-xgjvg\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:27:59.547926 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:59.547914    2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4314cef7-c274-488f-8609-f13e49d9b8b8-plugins-config-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:27:59.547926 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:27:59.547922    2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fada5537-42c7-491e-880b-856188c1ab8e-plugins-config-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:28:00.227986 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:00.227799    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-74690011-7c7f4bfdbd-5lj2s"
Jul 29 04:28:00.227986 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:00.227802    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-74690011-7c7f4bfdbd-5lj2s" event={"ID":"fada5537-42c7-491e-880b-856188c1ab8e","Type":"ContainerDied","Data":"c25948984a14b75e29c47fe0682562182742344e8bffbf4bc4b05a45de455131"}
Jul 29 04:28:00.227986 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:00.227965    2570 scope.go:117] "RemoveContainer" containerID="976be568438cc6e473546e718e8b39f01c47380455b38c2d3ed8a42274618473"
Jul 29 04:28:00.229423 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:00.229406    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-74690011-69f764786f-xzkxx"
Jul 29 04:28:00.229487 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:00.229445    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-74690011-69f764786f-xzkxx" event={"ID":"4314cef7-c274-488f-8609-f13e49d9b8b8","Type":"ContainerDied","Data":"354b053d28f00684ba041b8d5c41cc20ad4bb57c38cc041d80fc9ca6cf31fe26"}
Jul 29 04:28:00.231019 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:00.230987    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-74690011-74878c7b76-plgbl" event={"ID":"765426b8-3781-4988-9732-a041d5940e07","Type":"ContainerDied","Data":"f4660a6c4cd336ca393cac12dcccdfa78393d0d651f45788d9c5f038e57ccc4d"}
Jul 29 04:28:00.231019 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:00.231013    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-74690011-74878c7b76-plgbl"
Jul 29 04:28:00.238307 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:00.238268    2570 scope.go:117] "RemoveContainer" containerID="ceb505cb44ba54b6cc4eee26ecb40dc6724fc446370d422c1c7cef2b850917ad"
Jul 29 04:28:00.246451 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:00.246431    2570 scope.go:117] "RemoveContainer" containerID="e8dc98de22434e4dcd62982ac523576d69a5cd877f0c7291327f31241f958336"
Jul 29 04:28:00.256225 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:00.256205    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-74690011-7c7f4bfdbd-5lj2s"]
Jul 29 04:28:00.260084 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:00.260062    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-74690011-7c7f4bfdbd-5lj2s"]
Jul 29 04:28:00.270444 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:00.270425    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-74690011-69f764786f-xzkxx"]
Jul 29 04:28:00.273998 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:00.273974    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-74690011-69f764786f-xzkxx"]
Jul 29 04:28:00.284589 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:00.284563    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-74690011-74878c7b76-plgbl"]
Jul 29 04:28:00.287968 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:00.287947    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-74690011-74878c7b76-plgbl"]
Jul 29 04:28:01.835459 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:01.835421    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4314cef7-c274-488f-8609-f13e49d9b8b8" path="/var/lib/kubelet/pods/4314cef7-c274-488f-8609-f13e49d9b8b8/volumes"
Jul 29 04:28:01.836028 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:01.836009    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="765426b8-3781-4988-9732-a041d5940e07" path="/var/lib/kubelet/pods/765426b8-3781-4988-9732-a041d5940e07/volumes"
Jul 29 04:28:01.836388 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:01.836370    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fada5537-42c7-491e-880b-856188c1ab8e" path="/var/lib/kubelet/pods/fada5537-42c7-491e-880b-856188c1ab8e/volumes"
Jul 29 04:28:02.665468 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:02.665434    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4dcdbfe9-6f9b87c464-6jbrz"]
Jul 29 04:28:02.666099 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:02.666078    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fada5537-42c7-491e-880b-856188c1ab8e" containerName="payload-pre-processing"
Jul 29 04:28:02.666149 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:02.666105    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="fada5537-42c7-491e-880b-856188c1ab8e" containerName="payload-pre-processing"
Jul 29 04:28:02.666149 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:02.666117    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4314cef7-c274-488f-8609-f13e49d9b8b8" containerName="payload-processing"
Jul 29 04:28:02.666149 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:02.666126    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="4314cef7-c274-488f-8609-f13e49d9b8b8" containerName="payload-processing"
Jul 29 04:28:02.666249 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:02.666152    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="765426b8-3781-4988-9732-a041d5940e07" containerName="maas-api"
Jul 29 04:28:02.666249 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:02.666162    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="765426b8-3781-4988-9732-a041d5940e07" containerName="maas-api"
Jul 29 04:28:02.666311 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:02.666255    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="765426b8-3781-4988-9732-a041d5940e07" containerName="maas-api"
Jul 29 04:28:02.666311 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:02.666274    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="fada5537-42c7-491e-880b-856188c1ab8e" containerName="payload-pre-processing"
Jul 29 04:28:02.666311 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:02.666283    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="4314cef7-c274-488f-8609-f13e49d9b8b8" containerName="payload-processing"
Jul 29 04:28:02.674869 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:02.674842    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dcdbfe9-6f9b87c464-6jbrz"
Jul 29 04:28:02.676102 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:02.676077    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4dcdbfe9-6f9b87c464-6jbrz"]
Jul 29 04:28:02.677679 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:02.677655    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-4dcdbfe9-serving-cert\""
Jul 29 04:28:02.677805 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:02.677726    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 29 04:28:02.679011 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:02.678989    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-lh6p9\""
Jul 29 04:28:02.679186 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:02.679167    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 29 04:28:02.774641 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:02.774605    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa0f42b2-1947-401d-92c5-c52bf159c96c-openshift-service-ca\") pod \"maas-api-e2e-ait-4dcdbfe9-6f9b87c464-6jbrz\" (UID: \"aa0f42b2-1947-401d-92c5-c52bf159c96c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dcdbfe9-6f9b87c464-6jbrz"
Jul 29 04:28:02.774802 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:02.774660    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vlk9\" (UniqueName: \"kubernetes.io/projected/aa0f42b2-1947-401d-92c5-c52bf159c96c-kube-api-access-7vlk9\") pod \"maas-api-e2e-ait-4dcdbfe9-6f9b87c464-6jbrz\" (UID: \"aa0f42b2-1947-401d-92c5-c52bf159c96c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dcdbfe9-6f9b87c464-6jbrz"
Jul 29 04:28:02.774802 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:02.774731    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa0f42b2-1947-401d-92c5-c52bf159c96c-maas-api-tls\") pod \"maas-api-e2e-ait-4dcdbfe9-6f9b87c464-6jbrz\" (UID: \"aa0f42b2-1947-401d-92c5-c52bf159c96c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dcdbfe9-6f9b87c464-6jbrz"
Jul 29 04:28:02.875945 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:02.875914    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vlk9\" (UniqueName: \"kubernetes.io/projected/aa0f42b2-1947-401d-92c5-c52bf159c96c-kube-api-access-7vlk9\") pod \"maas-api-e2e-ait-4dcdbfe9-6f9b87c464-6jbrz\" (UID: \"aa0f42b2-1947-401d-92c5-c52bf159c96c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dcdbfe9-6f9b87c464-6jbrz"
Jul 29 04:28:02.876287 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:02.875952    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa0f42b2-1947-401d-92c5-c52bf159c96c-maas-api-tls\") pod \"maas-api-e2e-ait-4dcdbfe9-6f9b87c464-6jbrz\" (UID: \"aa0f42b2-1947-401d-92c5-c52bf159c96c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dcdbfe9-6f9b87c464-6jbrz"
Jul 29 04:28:02.876287 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:02.876005    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa0f42b2-1947-401d-92c5-c52bf159c96c-openshift-service-ca\") pod \"maas-api-e2e-ait-4dcdbfe9-6f9b87c464-6jbrz\" (UID: \"aa0f42b2-1947-401d-92c5-c52bf159c96c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dcdbfe9-6f9b87c464-6jbrz"
Jul 29 04:28:02.876600 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:02.876582    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa0f42b2-1947-401d-92c5-c52bf159c96c-openshift-service-ca\") pod \"maas-api-e2e-ait-4dcdbfe9-6f9b87c464-6jbrz\" (UID: \"aa0f42b2-1947-401d-92c5-c52bf159c96c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dcdbfe9-6f9b87c464-6jbrz"
Jul 29 04:28:02.878475 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:02.878457    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa0f42b2-1947-401d-92c5-c52bf159c96c-maas-api-tls\") pod \"maas-api-e2e-ait-4dcdbfe9-6f9b87c464-6jbrz\" (UID: \"aa0f42b2-1947-401d-92c5-c52bf159c96c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dcdbfe9-6f9b87c464-6jbrz"
Jul 29 04:28:02.888356 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:02.888338    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vlk9\" (UniqueName: \"kubernetes.io/projected/aa0f42b2-1947-401d-92c5-c52bf159c96c-kube-api-access-7vlk9\") pod \"maas-api-e2e-ait-4dcdbfe9-6f9b87c464-6jbrz\" (UID: \"aa0f42b2-1947-401d-92c5-c52bf159c96c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dcdbfe9-6f9b87c464-6jbrz"
Jul 29 04:28:02.987275 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:02.987207    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dcdbfe9-6f9b87c464-6jbrz"
Jul 29 04:28:03.131058 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:03.131035    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4dcdbfe9-6f9b87c464-6jbrz"]
Jul 29 04:28:03.133923 ip-10-0-135-138 kubenswrapper[2570]: W0729 04:28:03.133896    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa0f42b2_1947_401d_92c5_c52bf159c96c.slice/crio-8bf5e532b20913df46b10f9b99e001021298320f3f29830a59328aac5b5ee342 WatchSource:0}: Error finding container 8bf5e532b20913df46b10f9b99e001021298320f3f29830a59328aac5b5ee342: Status 404 returned error can't find the container with id 8bf5e532b20913df46b10f9b99e001021298320f3f29830a59328aac5b5ee342
Jul 29 04:28:03.249537 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:03.249458    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dcdbfe9-6f9b87c464-6jbrz" event={"ID":"aa0f42b2-1947-401d-92c5-c52bf159c96c","Type":"ContainerStarted","Data":"8bf5e532b20913df46b10f9b99e001021298320f3f29830a59328aac5b5ee342"}
Jul 29 04:28:03.397391 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:03.397355    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4dcdbfe9-5d94c847c6-v5d7m"]
Jul 29 04:28:03.402696 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:03.402679    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4dcdbfe9-5d94c847c6-v5d7m"
Jul 29 04:28:03.405336 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:03.405315    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-4dcdbfe9\""
Jul 29 04:28:03.405439 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:03.405367    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-4dcdbfe9-dockercfg-5v6vh\""
Jul 29 04:28:03.411777 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:03.411753    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4dcdbfe9-5d94c847c6-v5d7m"]
Jul 29 04:28:03.435043 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:03.435011    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4dcdbfe9-6d94d8484b-5jxlg"]
Jul 29 04:28:03.439847 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:03.439794    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4dcdbfe9-6d94d8484b-5jxlg"
Jul 29 04:28:03.445145 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:03.445122    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4dcdbfe9-6d94d8484b-5jxlg"]
Jul 29 04:28:03.481474 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:03.481447    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnqw7\" (UniqueName: \"kubernetes.io/projected/9c7e988e-ab93-46d9-9127-6128c8bd6705-kube-api-access-vnqw7\") pod \"payload-pre-processing-e2e-ait-4dcdbfe9-5d94c847c6-v5d7m\" (UID: \"9c7e988e-ab93-46d9-9127-6128c8bd6705\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4dcdbfe9-5d94c847c6-v5d7m"
Jul 29 04:28:03.481599 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:03.481562    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c7e988e-ab93-46d9-9127-6128c8bd6705-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4dcdbfe9-5d94c847c6-v5d7m\" (UID: \"9c7e988e-ab93-46d9-9127-6128c8bd6705\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4dcdbfe9-5d94c847c6-v5d7m"
Jul 29 04:28:03.582523 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:03.582496    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c7e988e-ab93-46d9-9127-6128c8bd6705-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4dcdbfe9-5d94c847c6-v5d7m\" (UID: \"9c7e988e-ab93-46d9-9127-6128c8bd6705\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4dcdbfe9-5d94c847c6-v5d7m"
Jul 29 04:28:03.582650 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:03.582531    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8979z\" (UniqueName: \"kubernetes.io/projected/08a1ca5c-a46c-4e2d-8ade-34941241fcb2-kube-api-access-8979z\") pod \"payload-processing-e2e-ait-4dcdbfe9-6d94d8484b-5jxlg\" (UID: \"08a1ca5c-a46c-4e2d-8ade-34941241fcb2\") " pod="openshift-ingress/payload-processing-e2e-ait-4dcdbfe9-6d94d8484b-5jxlg"
Jul 29 04:28:03.582650 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:03.582569    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnqw7\" (UniqueName: \"kubernetes.io/projected/9c7e988e-ab93-46d9-9127-6128c8bd6705-kube-api-access-vnqw7\") pod \"payload-pre-processing-e2e-ait-4dcdbfe9-5d94c847c6-v5d7m\" (UID: \"9c7e988e-ab93-46d9-9127-6128c8bd6705\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4dcdbfe9-5d94c847c6-v5d7m"
Jul 29 04:28:03.582650 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:03.582597    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08a1ca5c-a46c-4e2d-8ade-34941241fcb2-plugins-config-volume\") pod \"payload-processing-e2e-ait-4dcdbfe9-6d94d8484b-5jxlg\" (UID: \"08a1ca5c-a46c-4e2d-8ade-34941241fcb2\") " pod="openshift-ingress/payload-processing-e2e-ait-4dcdbfe9-6d94d8484b-5jxlg"
Jul 29 04:28:03.583188 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:03.583169    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c7e988e-ab93-46d9-9127-6128c8bd6705-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4dcdbfe9-5d94c847c6-v5d7m\" (UID: \"9c7e988e-ab93-46d9-9127-6128c8bd6705\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4dcdbfe9-5d94c847c6-v5d7m"
Jul 29 04:28:03.590386 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:03.590365    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnqw7\" (UniqueName: \"kubernetes.io/projected/9c7e988e-ab93-46d9-9127-6128c8bd6705-kube-api-access-vnqw7\") pod \"payload-pre-processing-e2e-ait-4dcdbfe9-5d94c847c6-v5d7m\" (UID: \"9c7e988e-ab93-46d9-9127-6128c8bd6705\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4dcdbfe9-5d94c847c6-v5d7m"
Jul 29 04:28:03.683536 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:03.683502    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08a1ca5c-a46c-4e2d-8ade-34941241fcb2-plugins-config-volume\") pod \"payload-processing-e2e-ait-4dcdbfe9-6d94d8484b-5jxlg\" (UID: \"08a1ca5c-a46c-4e2d-8ade-34941241fcb2\") " pod="openshift-ingress/payload-processing-e2e-ait-4dcdbfe9-6d94d8484b-5jxlg"
Jul 29 04:28:03.683696 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:03.683592    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8979z\" (UniqueName: \"kubernetes.io/projected/08a1ca5c-a46c-4e2d-8ade-34941241fcb2-kube-api-access-8979z\") pod \"payload-processing-e2e-ait-4dcdbfe9-6d94d8484b-5jxlg\" (UID: \"08a1ca5c-a46c-4e2d-8ade-34941241fcb2\") " pod="openshift-ingress/payload-processing-e2e-ait-4dcdbfe9-6d94d8484b-5jxlg"
Jul 29 04:28:03.684188 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:03.684170    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08a1ca5c-a46c-4e2d-8ade-34941241fcb2-plugins-config-volume\") pod \"payload-processing-e2e-ait-4dcdbfe9-6d94d8484b-5jxlg\" (UID: \"08a1ca5c-a46c-4e2d-8ade-34941241fcb2\") " pod="openshift-ingress/payload-processing-e2e-ait-4dcdbfe9-6d94d8484b-5jxlg"
Jul 29 04:28:03.691726 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:03.691708    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8979z\" (UniqueName: \"kubernetes.io/projected/08a1ca5c-a46c-4e2d-8ade-34941241fcb2-kube-api-access-8979z\") pod \"payload-processing-e2e-ait-4dcdbfe9-6d94d8484b-5jxlg\" (UID: \"08a1ca5c-a46c-4e2d-8ade-34941241fcb2\") " pod="openshift-ingress/payload-processing-e2e-ait-4dcdbfe9-6d94d8484b-5jxlg"
Jul 29 04:28:03.715621 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:03.715598    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4dcdbfe9-5d94c847c6-v5d7m"
Jul 29 04:28:03.750999 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:03.750972    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4dcdbfe9-6d94d8484b-5jxlg"
Jul 29 04:28:03.901412 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:03.901225    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4dcdbfe9-5d94c847c6-v5d7m"]
Jul 29 04:28:03.903391 ip-10-0-135-138 kubenswrapper[2570]: W0729 04:28:03.903358    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c7e988e_ab93_46d9_9127_6128c8bd6705.slice/crio-5f8bc803bb840cece5fc6facaa61c15b03272f978ae9f101c25f8935eb2c1a50 WatchSource:0}: Error finding container 5f8bc803bb840cece5fc6facaa61c15b03272f978ae9f101c25f8935eb2c1a50: Status 404 returned error can't find the container with id 5f8bc803bb840cece5fc6facaa61c15b03272f978ae9f101c25f8935eb2c1a50
Jul 29 04:28:04.134069 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:04.133998    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4dcdbfe9-6d94d8484b-5jxlg"]
Jul 29 04:28:04.137204 ip-10-0-135-138 kubenswrapper[2570]: W0729 04:28:04.137173    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08a1ca5c_a46c_4e2d_8ade_34941241fcb2.slice/crio-30910611048c7b21f73c226736824fad12e2d9c30a998a9d2b974e1ec7fc4f47 WatchSource:0}: Error finding container 30910611048c7b21f73c226736824fad12e2d9c30a998a9d2b974e1ec7fc4f47: Status 404 returned error can't find the container with id 30910611048c7b21f73c226736824fad12e2d9c30a998a9d2b974e1ec7fc4f47
Jul 29 04:28:04.256488 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:04.256418    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4dcdbfe9-6d94d8484b-5jxlg" event={"ID":"08a1ca5c-a46c-4e2d-8ade-34941241fcb2","Type":"ContainerStarted","Data":"15556ce37474b5e95a00982cbefe9dd0667c608b2d235d78113443b3a090c8d9"}
Jul 29 04:28:04.256660 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:04.256584    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-4dcdbfe9-6d94d8484b-5jxlg"
Jul 29 04:28:04.256660 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:04.256627    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4dcdbfe9-6d94d8484b-5jxlg" event={"ID":"08a1ca5c-a46c-4e2d-8ade-34941241fcb2","Type":"ContainerStarted","Data":"30910611048c7b21f73c226736824fad12e2d9c30a998a9d2b974e1ec7fc4f47"}
Jul 29 04:28:04.258022 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:04.258000    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dcdbfe9-6f9b87c464-6jbrz" event={"ID":"aa0f42b2-1947-401d-92c5-c52bf159c96c","Type":"ContainerStarted","Data":"095a15332915e9916633668501e6f9e282f0892bc029e1d915aec491822b75a0"}
Jul 29 04:28:04.258173 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:04.258046    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dcdbfe9-6f9b87c464-6jbrz"
Jul 29 04:28:04.259526 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:04.259503    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4dcdbfe9-5d94c847c6-v5d7m" event={"ID":"9c7e988e-ab93-46d9-9127-6128c8bd6705","Type":"ContainerStarted","Data":"5100a374bf92a41402b6bdd1618a7db348740f68add8637933f1fd0e15c38e02"}
Jul 29 04:28:04.259614 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:04.259547    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4dcdbfe9-5d94c847c6-v5d7m" event={"ID":"9c7e988e-ab93-46d9-9127-6128c8bd6705","Type":"ContainerStarted","Data":"5f8bc803bb840cece5fc6facaa61c15b03272f978ae9f101c25f8935eb2c1a50"}
Jul 29 04:28:04.259614 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:04.259572    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-4dcdbfe9-5d94c847c6-v5d7m"
Jul 29 04:28:04.275432 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:04.275390    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-4dcdbfe9-6d94d8484b-5jxlg" podStartSLOduration=1.275375633 podStartE2EDuration="1.275375633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:28:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:28:04.27353893 +0000 UTC m=+1949.003670018" watchObservedRunningTime="2026-07-29 04:28:04.275375633 +0000 UTC m=+1949.005506703"
Jul 29 04:28:04.288469 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:04.288430    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-4dcdbfe9-5d94c847c6-v5d7m" podStartSLOduration=1.288417841 podStartE2EDuration="1.288417841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:28:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:28:04.287571241 +0000 UTC m=+1949.017702332" watchObservedRunningTime="2026-07-29 04:28:04.288417841 +0000 UTC m=+1949.018548889"
Jul 29 04:28:04.304424 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:04.304386    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dcdbfe9-6f9b87c464-6jbrz" podStartSLOduration=1.7130302670000002 podStartE2EDuration="2.304374939s" totalImagesPullingTime="591.344672ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:28:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:28:04.302889074 +0000 UTC m=+1949.033020166" watchObservedRunningTime="2026-07-29 04:28:04.304374939 +0000 UTC m=+1949.034506009"
Jul 29 04:28:10.271203 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:10.271169    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dcdbfe9-6f9b87c464-6jbrz"
Jul 29 04:28:15.266567 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:15.266530    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-4dcdbfe9-5d94c847c6-v5d7m"
Jul 29 04:28:15.266998 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:15.266605    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-4dcdbfe9-6d94d8484b-5jxlg"
Jul 29 04:28:33.309677 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:33.309602    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4dcdbfe9-6f9b87c464-6jbrz"]
Jul 29 04:28:33.310251 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:33.309845    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dcdbfe9-6f9b87c464-6jbrz" podUID="aa0f42b2-1947-401d-92c5-c52bf159c96c" containerName="maas-api" containerID="cri-o://095a15332915e9916633668501e6f9e282f0892bc029e1d915aec491822b75a0" gracePeriod=30
Jul 29 04:28:33.357871 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:33.357842    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4dcdbfe9-6d94d8484b-5jxlg"]
Jul 29 04:28:33.358073 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:33.358050    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-4dcdbfe9-6d94d8484b-5jxlg" podUID="08a1ca5c-a46c-4e2d-8ade-34941241fcb2" containerName="payload-processing" containerID="cri-o://15556ce37474b5e95a00982cbefe9dd0667c608b2d235d78113443b3a090c8d9" gracePeriod=30
Jul 29 04:28:33.372939 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:33.370863    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4dcdbfe9-5d94c847c6-v5d7m"]
Jul 29 04:28:33.373510 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:33.373456    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-4dcdbfe9-5d94c847c6-v5d7m" podUID="9c7e988e-ab93-46d9-9127-6128c8bd6705" containerName="payload-pre-processing" containerID="cri-o://5100a374bf92a41402b6bdd1618a7db348740f68add8637933f1fd0e15c38e02" gracePeriod=30
Jul 29 04:28:33.640462 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:33.640433    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dcdbfe9-6f9b87c464-6jbrz"
Jul 29 04:28:33.650519 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:33.650497    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4dcdbfe9-6d94d8484b-5jxlg"
Jul 29 04:28:33.664916 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:33.664897    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4dcdbfe9-5d94c847c6-v5d7m"
Jul 29 04:28:33.776457 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:33.776426    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8979z\" (UniqueName: \"kubernetes.io/projected/08a1ca5c-a46c-4e2d-8ade-34941241fcb2-kube-api-access-8979z\") pod \"08a1ca5c-a46c-4e2d-8ade-34941241fcb2\" (UID: \"08a1ca5c-a46c-4e2d-8ade-34941241fcb2\") "
Jul 29 04:28:33.776619 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:33.776467    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vnqw7\" (UniqueName: \"kubernetes.io/projected/9c7e988e-ab93-46d9-9127-6128c8bd6705-kube-api-access-vnqw7\") pod \"9c7e988e-ab93-46d9-9127-6128c8bd6705\" (UID: \"9c7e988e-ab93-46d9-9127-6128c8bd6705\") "
Jul 29 04:28:33.776619 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:33.776503    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa0f42b2-1947-401d-92c5-c52bf159c96c-openshift-service-ca\") pod \"aa0f42b2-1947-401d-92c5-c52bf159c96c\" (UID: \"aa0f42b2-1947-401d-92c5-c52bf159c96c\") "
Jul 29 04:28:33.776619 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:33.776547    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7vlk9\" (UniqueName: \"kubernetes.io/projected/aa0f42b2-1947-401d-92c5-c52bf159c96c-kube-api-access-7vlk9\") pod \"aa0f42b2-1947-401d-92c5-c52bf159c96c\" (UID: \"aa0f42b2-1947-401d-92c5-c52bf159c96c\") "
Jul 29 04:28:33.776619 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:33.776578    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c7e988e-ab93-46d9-9127-6128c8bd6705-plugins-config-volume\") pod \"9c7e988e-ab93-46d9-9127-6128c8bd6705\" (UID: \"9c7e988e-ab93-46d9-9127-6128c8bd6705\") "
Jul 29 04:28:33.776846 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:33.776623    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa0f42b2-1947-401d-92c5-c52bf159c96c-maas-api-tls\") pod \"aa0f42b2-1947-401d-92c5-c52bf159c96c\" (UID: \"aa0f42b2-1947-401d-92c5-c52bf159c96c\") "
Jul 29 04:28:33.776846 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:33.776648    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08a1ca5c-a46c-4e2d-8ade-34941241fcb2-plugins-config-volume\") pod \"08a1ca5c-a46c-4e2d-8ade-34941241fcb2\" (UID: \"08a1ca5c-a46c-4e2d-8ade-34941241fcb2\") "
Jul 29 04:28:33.776971 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:33.776944    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa0f42b2-1947-401d-92c5-c52bf159c96c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "aa0f42b2-1947-401d-92c5-c52bf159c96c" (UID: "aa0f42b2-1947-401d-92c5-c52bf159c96c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 04:28:33.777204 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:33.777154    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c7e988e-ab93-46d9-9127-6128c8bd6705-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9c7e988e-ab93-46d9-9127-6128c8bd6705" (UID: "9c7e988e-ab93-46d9-9127-6128c8bd6705"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 04:28:33.777326 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:33.777203    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08a1ca5c-a46c-4e2d-8ade-34941241fcb2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "08a1ca5c-a46c-4e2d-8ade-34941241fcb2" (UID: "08a1ca5c-a46c-4e2d-8ade-34941241fcb2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 04:28:33.778803 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:33.778775    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c7e988e-ab93-46d9-9127-6128c8bd6705-kube-api-access-vnqw7" (OuterVolumeSpecName: "kube-api-access-vnqw7") pod "9c7e988e-ab93-46d9-9127-6128c8bd6705" (UID: "9c7e988e-ab93-46d9-9127-6128c8bd6705"). InnerVolumeSpecName "kube-api-access-vnqw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 04:28:33.778943 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:33.778925    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa0f42b2-1947-401d-92c5-c52bf159c96c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "aa0f42b2-1947-401d-92c5-c52bf159c96c" (UID: "aa0f42b2-1947-401d-92c5-c52bf159c96c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 04:28:33.779140 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:33.779126    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa0f42b2-1947-401d-92c5-c52bf159c96c-kube-api-access-7vlk9" (OuterVolumeSpecName: "kube-api-access-7vlk9") pod "aa0f42b2-1947-401d-92c5-c52bf159c96c" (UID: "aa0f42b2-1947-401d-92c5-c52bf159c96c"). InnerVolumeSpecName "kube-api-access-7vlk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 04:28:33.779202 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:33.779187    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08a1ca5c-a46c-4e2d-8ade-34941241fcb2-kube-api-access-8979z" (OuterVolumeSpecName: "kube-api-access-8979z") pod "08a1ca5c-a46c-4e2d-8ade-34941241fcb2" (UID: "08a1ca5c-a46c-4e2d-8ade-34941241fcb2"). InnerVolumeSpecName "kube-api-access-8979z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 04:28:33.877741 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:33.877713    2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08a1ca5c-a46c-4e2d-8ade-34941241fcb2-plugins-config-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:28:33.877741 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:33.877737    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8979z\" (UniqueName: \"kubernetes.io/projected/08a1ca5c-a46c-4e2d-8ade-34941241fcb2-kube-api-access-8979z\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:28:33.877741 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:33.877746    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnqw7\" (UniqueName: \"kubernetes.io/projected/9c7e988e-ab93-46d9-9127-6128c8bd6705-kube-api-access-vnqw7\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:28:33.877929 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:33.877755    2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa0f42b2-1947-401d-92c5-c52bf159c96c-openshift-service-ca\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:28:33.877929 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:33.877763    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vlk9\" (UniqueName: \"kubernetes.io/projected/aa0f42b2-1947-401d-92c5-c52bf159c96c-kube-api-access-7vlk9\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:28:33.877929 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:33.877771    2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c7e988e-ab93-46d9-9127-6128c8bd6705-plugins-config-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:28:33.877929 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:33.877779    2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa0f42b2-1947-401d-92c5-c52bf159c96c-maas-api-tls\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:28:34.412057 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:34.412025    2570 generic.go:356] "Generic (PLEG): container finished" podID="aa0f42b2-1947-401d-92c5-c52bf159c96c" containerID="095a15332915e9916633668501e6f9e282f0892bc029e1d915aec491822b75a0" exitCode=0
Jul 29 04:28:34.412523 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:34.412074    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dcdbfe9-6f9b87c464-6jbrz"
Jul 29 04:28:34.412523 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:34.412126    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dcdbfe9-6f9b87c464-6jbrz" event={"ID":"aa0f42b2-1947-401d-92c5-c52bf159c96c","Type":"ContainerDied","Data":"095a15332915e9916633668501e6f9e282f0892bc029e1d915aec491822b75a0"}
Jul 29 04:28:34.412523 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:34.412188    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dcdbfe9-6f9b87c464-6jbrz" event={"ID":"aa0f42b2-1947-401d-92c5-c52bf159c96c","Type":"ContainerDied","Data":"8bf5e532b20913df46b10f9b99e001021298320f3f29830a59328aac5b5ee342"}
Jul 29 04:28:34.412523 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:34.412218    2570 scope.go:117] "RemoveContainer" containerID="095a15332915e9916633668501e6f9e282f0892bc029e1d915aec491822b75a0"
Jul 29 04:28:34.413628 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:34.413608    2570 generic.go:356] "Generic (PLEG): container finished" podID="9c7e988e-ab93-46d9-9127-6128c8bd6705" containerID="5100a374bf92a41402b6bdd1618a7db348740f68add8637933f1fd0e15c38e02" exitCode=0
Jul 29 04:28:34.413712 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:34.413639    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4dcdbfe9-5d94c847c6-v5d7m" event={"ID":"9c7e988e-ab93-46d9-9127-6128c8bd6705","Type":"ContainerDied","Data":"5100a374bf92a41402b6bdd1618a7db348740f68add8637933f1fd0e15c38e02"}
Jul 29 04:28:34.413712 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:34.413663    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4dcdbfe9-5d94c847c6-v5d7m"
Jul 29 04:28:34.413712 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:34.413689    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4dcdbfe9-5d94c847c6-v5d7m" event={"ID":"9c7e988e-ab93-46d9-9127-6128c8bd6705","Type":"ContainerDied","Data":"5f8bc803bb840cece5fc6facaa61c15b03272f978ae9f101c25f8935eb2c1a50"}
Jul 29 04:28:34.415182 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:34.415146    2570 generic.go:356] "Generic (PLEG): container finished" podID="08a1ca5c-a46c-4e2d-8ade-34941241fcb2" containerID="15556ce37474b5e95a00982cbefe9dd0667c608b2d235d78113443b3a090c8d9" exitCode=0
Jul 29 04:28:34.415321 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:34.415183    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4dcdbfe9-6d94d8484b-5jxlg" event={"ID":"08a1ca5c-a46c-4e2d-8ade-34941241fcb2","Type":"ContainerDied","Data":"15556ce37474b5e95a00982cbefe9dd0667c608b2d235d78113443b3a090c8d9"}
Jul 29 04:28:34.415321 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:34.415218    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4dcdbfe9-6d94d8484b-5jxlg" event={"ID":"08a1ca5c-a46c-4e2d-8ade-34941241fcb2","Type":"ContainerDied","Data":"30910611048c7b21f73c226736824fad12e2d9c30a998a9d2b974e1ec7fc4f47"}
Jul 29 04:28:34.415321 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:34.415273    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4dcdbfe9-6d94d8484b-5jxlg"
Jul 29 04:28:34.423112 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:34.423093    2570 scope.go:117] "RemoveContainer" containerID="095a15332915e9916633668501e6f9e282f0892bc029e1d915aec491822b75a0"
Jul 29 04:28:34.423375 ip-10-0-135-138 kubenswrapper[2570]: E0729 04:28:34.423357    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"095a15332915e9916633668501e6f9e282f0892bc029e1d915aec491822b75a0\": container with ID starting with 095a15332915e9916633668501e6f9e282f0892bc029e1d915aec491822b75a0 not found: ID does not exist" containerID="095a15332915e9916633668501e6f9e282f0892bc029e1d915aec491822b75a0"
Jul 29 04:28:34.423431 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:34.423385    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"095a15332915e9916633668501e6f9e282f0892bc029e1d915aec491822b75a0"} err="failed to get container status \"095a15332915e9916633668501e6f9e282f0892bc029e1d915aec491822b75a0\": rpc error: code = NotFound desc = could not find container \"095a15332915e9916633668501e6f9e282f0892bc029e1d915aec491822b75a0\": container with ID starting with 095a15332915e9916633668501e6f9e282f0892bc029e1d915aec491822b75a0 not found: ID does not exist"
Jul 29 04:28:34.423431 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:34.423402    2570 scope.go:117] "RemoveContainer" containerID="5100a374bf92a41402b6bdd1618a7db348740f68add8637933f1fd0e15c38e02"
Jul 29 04:28:34.432149 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:34.432128    2570 scope.go:117] "RemoveContainer" containerID="5100a374bf92a41402b6bdd1618a7db348740f68add8637933f1fd0e15c38e02"
Jul 29 04:28:34.432316 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:34.432294    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4dcdbfe9-6f9b87c464-6jbrz"]
Jul 29 04:28:34.432462 ip-10-0-135-138 kubenswrapper[2570]: E0729 04:28:34.432444    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5100a374bf92a41402b6bdd1618a7db348740f68add8637933f1fd0e15c38e02\": container with ID starting with 5100a374bf92a41402b6bdd1618a7db348740f68add8637933f1fd0e15c38e02 not found: ID does not exist" containerID="5100a374bf92a41402b6bdd1618a7db348740f68add8637933f1fd0e15c38e02"
Jul 29 04:28:34.432545 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:34.432473    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5100a374bf92a41402b6bdd1618a7db348740f68add8637933f1fd0e15c38e02"} err="failed to get container status \"5100a374bf92a41402b6bdd1618a7db348740f68add8637933f1fd0e15c38e02\": rpc error: code = NotFound desc = could not find container \"5100a374bf92a41402b6bdd1618a7db348740f68add8637933f1fd0e15c38e02\": container with ID starting with 5100a374bf92a41402b6bdd1618a7db348740f68add8637933f1fd0e15c38e02 not found: ID does not exist"
Jul 29 04:28:34.432545 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:34.432497    2570 scope.go:117] "RemoveContainer" containerID="15556ce37474b5e95a00982cbefe9dd0667c608b2d235d78113443b3a090c8d9"
Jul 29 04:28:34.435399 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:34.435375    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4dcdbfe9-6f9b87c464-6jbrz"]
Jul 29 04:28:34.440909 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:34.440892    2570 scope.go:117] "RemoveContainer" containerID="15556ce37474b5e95a00982cbefe9dd0667c608b2d235d78113443b3a090c8d9"
Jul 29 04:28:34.441158 ip-10-0-135-138 kubenswrapper[2570]: E0729 04:28:34.441139    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15556ce37474b5e95a00982cbefe9dd0667c608b2d235d78113443b3a090c8d9\": container with ID starting with 15556ce37474b5e95a00982cbefe9dd0667c608b2d235d78113443b3a090c8d9 not found: ID does not exist" containerID="15556ce37474b5e95a00982cbefe9dd0667c608b2d235d78113443b3a090c8d9"
Jul 29 04:28:34.441210 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:34.441165    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15556ce37474b5e95a00982cbefe9dd0667c608b2d235d78113443b3a090c8d9"} err="failed to get container status \"15556ce37474b5e95a00982cbefe9dd0667c608b2d235d78113443b3a090c8d9\": rpc error: code = NotFound desc = could not find container \"15556ce37474b5e95a00982cbefe9dd0667c608b2d235d78113443b3a090c8d9\": container with ID starting with 15556ce37474b5e95a00982cbefe9dd0667c608b2d235d78113443b3a090c8d9 not found: ID does not exist"
Jul 29 04:28:34.451212 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:34.451188    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4dcdbfe9-5d94c847c6-v5d7m"]
Jul 29 04:28:34.457667 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:34.457647    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4dcdbfe9-5d94c847c6-v5d7m"]
Jul 29 04:28:34.467889 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:34.467865    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4dcdbfe9-6d94d8484b-5jxlg"]
Jul 29 04:28:34.469672 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:34.469652    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4dcdbfe9-6d94d8484b-5jxlg"]
Jul 29 04:28:35.836111 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:35.836079    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08a1ca5c-a46c-4e2d-8ade-34941241fcb2" path="/var/lib/kubelet/pods/08a1ca5c-a46c-4e2d-8ade-34941241fcb2/volumes"
Jul 29 04:28:35.836477 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:35.836419    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c7e988e-ab93-46d9-9127-6128c8bd6705" path="/var/lib/kubelet/pods/9c7e988e-ab93-46d9-9127-6128c8bd6705/volumes"
Jul 29 04:28:35.836751 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:35.836738    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa0f42b2-1947-401d-92c5-c52bf159c96c" path="/var/lib/kubelet/pods/aa0f42b2-1947-401d-92c5-c52bf159c96c/volumes"
Jul 29 04:28:37.843328 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:37.843295    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a77d4166-b47d659bb-55qg5"]
Jul 29 04:28:37.843940 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:37.843920    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c7e988e-ab93-46d9-9127-6128c8bd6705" containerName="payload-pre-processing"
Jul 29 04:28:37.843991 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:37.843944    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c7e988e-ab93-46d9-9127-6128c8bd6705" containerName="payload-pre-processing"
Jul 29 04:28:37.843991 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:37.843967    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa0f42b2-1947-401d-92c5-c52bf159c96c" containerName="maas-api"
Jul 29 04:28:37.843991 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:37.843977    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa0f42b2-1947-401d-92c5-c52bf159c96c" containerName="maas-api"
Jul 29 04:28:37.844081 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:37.843997    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08a1ca5c-a46c-4e2d-8ade-34941241fcb2" containerName="payload-processing"
Jul 29 04:28:37.844081 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:37.844006    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="08a1ca5c-a46c-4e2d-8ade-34941241fcb2" containerName="payload-processing"
Jul 29 04:28:37.844157 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:37.844095    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c7e988e-ab93-46d9-9127-6128c8bd6705" containerName="payload-pre-processing"
Jul 29 04:28:37.844157 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:37.844112    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="08a1ca5c-a46c-4e2d-8ade-34941241fcb2" containerName="payload-processing"
Jul 29 04:28:37.844157 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:37.844124    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa0f42b2-1947-401d-92c5-c52bf159c96c" containerName="maas-api"
Jul 29 04:28:37.847536 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:37.847519    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a77d4166-b47d659bb-55qg5"
Jul 29 04:28:37.850276 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:37.850250    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-a77d4166-serving-cert\""
Jul 29 04:28:37.850398 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:37.850270    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 29 04:28:37.850398 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:37.850318    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-lh6p9\""
Jul 29 04:28:37.851279 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:37.851262    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 29 04:28:37.856291 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:37.856270    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a77d4166-b47d659bb-55qg5"]
Jul 29 04:28:37.918230 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:37.918202    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef50925c-5b65-4cd1-bf28-8334af42a7c4-openshift-service-ca\") pod \"maas-api-e2e-derive-a77d4166-b47d659bb-55qg5\" (UID: \"ef50925c-5b65-4cd1-bf28-8334af42a7c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a77d4166-b47d659bb-55qg5"
Jul 29 04:28:37.918406 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:37.918272    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkfzb\" (UniqueName: \"kubernetes.io/projected/ef50925c-5b65-4cd1-bf28-8334af42a7c4-kube-api-access-xkfzb\") pod \"maas-api-e2e-derive-a77d4166-b47d659bb-55qg5\" (UID: \"ef50925c-5b65-4cd1-bf28-8334af42a7c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a77d4166-b47d659bb-55qg5"
Jul 29 04:28:37.918406 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:37.918309    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef50925c-5b65-4cd1-bf28-8334af42a7c4-maas-api-tls\") pod \"maas-api-e2e-derive-a77d4166-b47d659bb-55qg5\" (UID: \"ef50925c-5b65-4cd1-bf28-8334af42a7c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a77d4166-b47d659bb-55qg5"
Jul 29 04:28:38.019355 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:38.019321    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef50925c-5b65-4cd1-bf28-8334af42a7c4-openshift-service-ca\") pod \"maas-api-e2e-derive-a77d4166-b47d659bb-55qg5\" (UID: \"ef50925c-5b65-4cd1-bf28-8334af42a7c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a77d4166-b47d659bb-55qg5"
Jul 29 04:28:38.019493 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:38.019434    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkfzb\" (UniqueName: \"kubernetes.io/projected/ef50925c-5b65-4cd1-bf28-8334af42a7c4-kube-api-access-xkfzb\") pod \"maas-api-e2e-derive-a77d4166-b47d659bb-55qg5\" (UID: \"ef50925c-5b65-4cd1-bf28-8334af42a7c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a77d4166-b47d659bb-55qg5"
Jul 29 04:28:38.019493 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:38.019468    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef50925c-5b65-4cd1-bf28-8334af42a7c4-maas-api-tls\") pod \"maas-api-e2e-derive-a77d4166-b47d659bb-55qg5\" (UID: \"ef50925c-5b65-4cd1-bf28-8334af42a7c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a77d4166-b47d659bb-55qg5"
Jul 29 04:28:38.019603 ip-10-0-135-138 kubenswrapper[2570]: E0729 04:28:38.019589    2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-a77d4166-serving-cert: secret "maas-api-e2e-derive-a77d4166-serving-cert" not found
Jul 29 04:28:38.019672 ip-10-0-135-138 kubenswrapper[2570]: E0729 04:28:38.019661    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef50925c-5b65-4cd1-bf28-8334af42a7c4-maas-api-tls podName:ef50925c-5b65-4cd1-bf28-8334af42a7c4 nodeName:}" failed. No retries permitted until 2026-07-29 04:28:38.519634855 +0000 UTC m=+1983.249765904 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ef50925c-5b65-4cd1-bf28-8334af42a7c4-maas-api-tls") pod "maas-api-e2e-derive-a77d4166-b47d659bb-55qg5" (UID: "ef50925c-5b65-4cd1-bf28-8334af42a7c4") : secret "maas-api-e2e-derive-a77d4166-serving-cert" not found
Jul 29 04:28:38.020057 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:38.020040    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef50925c-5b65-4cd1-bf28-8334af42a7c4-openshift-service-ca\") pod \"maas-api-e2e-derive-a77d4166-b47d659bb-55qg5\" (UID: \"ef50925c-5b65-4cd1-bf28-8334af42a7c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a77d4166-b47d659bb-55qg5"
Jul 29 04:28:38.028538 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:38.028515    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkfzb\" (UniqueName: \"kubernetes.io/projected/ef50925c-5b65-4cd1-bf28-8334af42a7c4-kube-api-access-xkfzb\") pod \"maas-api-e2e-derive-a77d4166-b47d659bb-55qg5\" (UID: \"ef50925c-5b65-4cd1-bf28-8334af42a7c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a77d4166-b47d659bb-55qg5"
Jul 29 04:28:38.523801 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:38.523769    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef50925c-5b65-4cd1-bf28-8334af42a7c4-maas-api-tls\") pod \"maas-api-e2e-derive-a77d4166-b47d659bb-55qg5\" (UID: \"ef50925c-5b65-4cd1-bf28-8334af42a7c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a77d4166-b47d659bb-55qg5"
Jul 29 04:28:38.526585 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:38.526556    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef50925c-5b65-4cd1-bf28-8334af42a7c4-maas-api-tls\") pod \"maas-api-e2e-derive-a77d4166-b47d659bb-55qg5\" (UID: \"ef50925c-5b65-4cd1-bf28-8334af42a7c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a77d4166-b47d659bb-55qg5"
Jul 29 04:28:38.667485 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:38.667451    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a77d4166-5c56bc9676-5cdqk"]
Jul 29 04:28:38.671161 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:38.671139    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a77d4166-5c56bc9676-5cdqk"
Jul 29 04:28:38.674220 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:38.674197    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-a77d4166-dockercfg-jxbpl\""
Jul 29 04:28:38.674606 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:38.674585    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-a77d4166\""
Jul 29 04:28:38.681814 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:38.681790    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a77d4166-5c56bc9676-5cdqk"]
Jul 29 04:28:38.758654 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:38.758624    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a77d4166-b47d659bb-55qg5"
Jul 29 04:28:38.792458 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:38.792390    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a77d4166-6bd7bb48cd-xsxxd"]
Jul 29 04:28:38.796813 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:38.796794    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-a77d4166-6bd7bb48cd-xsxxd"
Jul 29 04:28:38.805549 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:38.805525    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a77d4166-6bd7bb48cd-xsxxd"]
Jul 29 04:28:38.827742 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:38.827712    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wsqj\" (UniqueName: \"kubernetes.io/projected/cdd4d96c-9cba-4b7e-b846-b796378a567a-kube-api-access-7wsqj\") pod \"payload-pre-processing-e2e-derive-a77d4166-5c56bc9676-5cdqk\" (UID: \"cdd4d96c-9cba-4b7e-b846-b796378a567a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a77d4166-5c56bc9676-5cdqk"
Jul 29 04:28:38.827912 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:38.827766    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdd4d96c-9cba-4b7e-b846-b796378a567a-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-a77d4166-5c56bc9676-5cdqk\" (UID: \"cdd4d96c-9cba-4b7e-b846-b796378a567a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a77d4166-5c56bc9676-5cdqk"
Jul 29 04:28:38.898415 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:38.898375    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a77d4166-b47d659bb-55qg5"]
Jul 29 04:28:38.901476 ip-10-0-135-138 kubenswrapper[2570]: W0729 04:28:38.901450    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef50925c_5b65_4cd1_bf28_8334af42a7c4.slice/crio-6579ffebd9ee9db1f97266f7456e72c196fec56e6cb8afb4e8e691e1261abae6 WatchSource:0}: Error finding container 6579ffebd9ee9db1f97266f7456e72c196fec56e6cb8afb4e8e691e1261abae6: Status 404 returned error can't find the container with id 6579ffebd9ee9db1f97266f7456e72c196fec56e6cb8afb4e8e691e1261abae6
Jul 29 04:28:38.929116 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:38.929090    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5223910-1215-481d-b5a0-0e36169a8be3-plugins-config-volume\") pod \"payload-processing-e2e-derive-a77d4166-6bd7bb48cd-xsxxd\" (UID: \"c5223910-1215-481d-b5a0-0e36169a8be3\") " pod="openshift-ingress/payload-processing-e2e-derive-a77d4166-6bd7bb48cd-xsxxd"
Jul 29 04:28:38.929220 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:38.929156    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvxbd\" (UniqueName: \"kubernetes.io/projected/c5223910-1215-481d-b5a0-0e36169a8be3-kube-api-access-cvxbd\") pod \"payload-processing-e2e-derive-a77d4166-6bd7bb48cd-xsxxd\" (UID: \"c5223910-1215-481d-b5a0-0e36169a8be3\") " pod="openshift-ingress/payload-processing-e2e-derive-a77d4166-6bd7bb48cd-xsxxd"
Jul 29 04:28:38.929220 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:38.929184    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wsqj\" (UniqueName: \"kubernetes.io/projected/cdd4d96c-9cba-4b7e-b846-b796378a567a-kube-api-access-7wsqj\") pod \"payload-pre-processing-e2e-derive-a77d4166-5c56bc9676-5cdqk\" (UID: \"cdd4d96c-9cba-4b7e-b846-b796378a567a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a77d4166-5c56bc9676-5cdqk"
Jul 29 04:28:38.929326 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:38.929223    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdd4d96c-9cba-4b7e-b846-b796378a567a-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-a77d4166-5c56bc9676-5cdqk\" (UID: \"cdd4d96c-9cba-4b7e-b846-b796378a567a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a77d4166-5c56bc9676-5cdqk"
Jul 29 04:28:38.929870 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:38.929855    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdd4d96c-9cba-4b7e-b846-b796378a567a-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-a77d4166-5c56bc9676-5cdqk\" (UID: \"cdd4d96c-9cba-4b7e-b846-b796378a567a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a77d4166-5c56bc9676-5cdqk"
Jul 29 04:28:38.937372 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:38.937355    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wsqj\" (UniqueName: \"kubernetes.io/projected/cdd4d96c-9cba-4b7e-b846-b796378a567a-kube-api-access-7wsqj\") pod \"payload-pre-processing-e2e-derive-a77d4166-5c56bc9676-5cdqk\" (UID: \"cdd4d96c-9cba-4b7e-b846-b796378a567a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a77d4166-5c56bc9676-5cdqk"
Jul 29 04:28:38.983339 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:38.983321    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a77d4166-5c56bc9676-5cdqk"
Jul 29 04:28:39.030330 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:39.030291    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5223910-1215-481d-b5a0-0e36169a8be3-plugins-config-volume\") pod \"payload-processing-e2e-derive-a77d4166-6bd7bb48cd-xsxxd\" (UID: \"c5223910-1215-481d-b5a0-0e36169a8be3\") " pod="openshift-ingress/payload-processing-e2e-derive-a77d4166-6bd7bb48cd-xsxxd"
Jul 29 04:28:39.030484 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:39.030353    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvxbd\" (UniqueName: \"kubernetes.io/projected/c5223910-1215-481d-b5a0-0e36169a8be3-kube-api-access-cvxbd\") pod \"payload-processing-e2e-derive-a77d4166-6bd7bb48cd-xsxxd\" (UID: \"c5223910-1215-481d-b5a0-0e36169a8be3\") " pod="openshift-ingress/payload-processing-e2e-derive-a77d4166-6bd7bb48cd-xsxxd"
Jul 29 04:28:39.031128 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:39.031104    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5223910-1215-481d-b5a0-0e36169a8be3-plugins-config-volume\") pod \"payload-processing-e2e-derive-a77d4166-6bd7bb48cd-xsxxd\" (UID: \"c5223910-1215-481d-b5a0-0e36169a8be3\") " pod="openshift-ingress/payload-processing-e2e-derive-a77d4166-6bd7bb48cd-xsxxd"
Jul 29 04:28:39.038352 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:39.038285    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvxbd\" (UniqueName: \"kubernetes.io/projected/c5223910-1215-481d-b5a0-0e36169a8be3-kube-api-access-cvxbd\") pod \"payload-processing-e2e-derive-a77d4166-6bd7bb48cd-xsxxd\" (UID: \"c5223910-1215-481d-b5a0-0e36169a8be3\") " pod="openshift-ingress/payload-processing-e2e-derive-a77d4166-6bd7bb48cd-xsxxd"
Jul 29 04:28:39.109402 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:39.109377    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-a77d4166-6bd7bb48cd-xsxxd"
Jul 29 04:28:39.114522 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:39.114498    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a77d4166-5c56bc9676-5cdqk"]
Jul 29 04:28:39.120117 ip-10-0-135-138 kubenswrapper[2570]: W0729 04:28:39.120094    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdd4d96c_9cba_4b7e_b846_b796378a567a.slice/crio-e30c50c0aac81c6d914665452d38cc9fd43e66a7b8a1b26f4438f87309029c69 WatchSource:0}: Error finding container e30c50c0aac81c6d914665452d38cc9fd43e66a7b8a1b26f4438f87309029c69: Status 404 returned error can't find the container with id e30c50c0aac81c6d914665452d38cc9fd43e66a7b8a1b26f4438f87309029c69
Jul 29 04:28:39.252058 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:39.251966    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a77d4166-6bd7bb48cd-xsxxd"]
Jul 29 04:28:39.254244 ip-10-0-135-138 kubenswrapper[2570]: W0729 04:28:39.254220    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5223910_1215_481d_b5a0_0e36169a8be3.slice/crio-03fbf76128c2902f8be34a6fafe15cb4730667f518937caecd04f432862843a1 WatchSource:0}: Error finding container 03fbf76128c2902f8be34a6fafe15cb4730667f518937caecd04f432862843a1: Status 404 returned error can't find the container with id 03fbf76128c2902f8be34a6fafe15cb4730667f518937caecd04f432862843a1
Jul 29 04:28:39.450976 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:39.450937    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a77d4166-b47d659bb-55qg5" event={"ID":"ef50925c-5b65-4cd1-bf28-8334af42a7c4","Type":"ContainerStarted","Data":"75ac7c849643f24fec392e2a46e8836db3662dbde969a2f8138ad3f2a6cc0b6b"}
Jul 29 04:28:39.451146 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:39.450999    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a77d4166-b47d659bb-55qg5"
Jul 29 04:28:39.451146 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:39.451019    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a77d4166-b47d659bb-55qg5" event={"ID":"ef50925c-5b65-4cd1-bf28-8334af42a7c4","Type":"ContainerStarted","Data":"6579ffebd9ee9db1f97266f7456e72c196fec56e6cb8afb4e8e691e1261abae6"}
Jul 29 04:28:39.452469 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:39.452443    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a77d4166-5c56bc9676-5cdqk" event={"ID":"cdd4d96c-9cba-4b7e-b846-b796378a567a","Type":"ContainerStarted","Data":"492ecdf505d9b52ec4eaea898df6f18bb285729b2b87bc7a558f0ac5f327b533"}
Jul 29 04:28:39.452614 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:39.452487    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a77d4166-5c56bc9676-5cdqk" event={"ID":"cdd4d96c-9cba-4b7e-b846-b796378a567a","Type":"ContainerStarted","Data":"e30c50c0aac81c6d914665452d38cc9fd43e66a7b8a1b26f4438f87309029c69"}
Jul 29 04:28:39.452614 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:39.452535    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-a77d4166-5c56bc9676-5cdqk"
Jul 29 04:28:39.454060 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:39.454034    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-a77d4166-6bd7bb48cd-xsxxd" event={"ID":"c5223910-1215-481d-b5a0-0e36169a8be3","Type":"ContainerStarted","Data":"3ebd8b5e3b8ca06ffd4338251f5aae8285f4c3b885fd0cff836002819f69ff80"}
Jul 29 04:28:39.454155 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:39.454080    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-a77d4166-6bd7bb48cd-xsxxd" event={"ID":"c5223910-1215-481d-b5a0-0e36169a8be3","Type":"ContainerStarted","Data":"03fbf76128c2902f8be34a6fafe15cb4730667f518937caecd04f432862843a1"}
Jul 29 04:28:39.454155 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:39.454108    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-a77d4166-6bd7bb48cd-xsxxd"
Jul 29 04:28:39.467521 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:39.467483    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a77d4166-b47d659bb-55qg5" podStartSLOduration=2.071862378 podStartE2EDuration="2.46746845s" totalImagesPullingTime="395.606072ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:28:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:28:39.465851899 +0000 UTC m=+1984.195982996" watchObservedRunningTime="2026-07-29 04:28:39.46746845 +0000 UTC m=+1984.197599520"
Jul 29 04:28:39.482507 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:39.482467    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-a77d4166-6bd7bb48cd-xsxxd" podStartSLOduration=1.482455154 podStartE2EDuration="1.482455154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:28:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:28:39.48061722 +0000 UTC m=+1984.210748296" watchObservedRunningTime="2026-07-29 04:28:39.482455154 +0000 UTC m=+1984.212586224"
Jul 29 04:28:39.495213 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:39.495178    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-a77d4166-5c56bc9676-5cdqk" podStartSLOduration=1.495166878 podStartE2EDuration="1.495166878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:28:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:28:39.494611936 +0000 UTC m=+1984.224743026" watchObservedRunningTime="2026-07-29 04:28:39.495166878 +0000 UTC m=+1984.225297948"
Jul 29 04:28:45.464933 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:45.464900    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a77d4166-b47d659bb-55qg5"
Jul 29 04:28:50.461172 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:50.461138    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-derive-a77d4166-5c56bc9676-5cdqk" podUID="cdd4d96c-9cba-4b7e-b846-b796378a567a" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.52:9004: connect: connection refused"
Jul 29 04:28:50.461573 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:28:50.461555    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-a77d4166-6bd7bb48cd-xsxxd"
Jul 29 04:29:00.460403 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:29:00.460373    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-a77d4166-5c56bc9676-5cdqk"
Jul 29 04:29:08.374737 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:29:08.374701    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a77d4166-b47d659bb-55qg5"]
Jul 29 04:29:08.375337 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:29:08.374938    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a77d4166-b47d659bb-55qg5" podUID="ef50925c-5b65-4cd1-bf28-8334af42a7c4" containerName="maas-api" containerID="cri-o://75ac7c849643f24fec392e2a46e8836db3662dbde969a2f8138ad3f2a6cc0b6b" gracePeriod=30
Jul 29 04:29:08.427513 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:29:08.427479    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a77d4166-6bd7bb48cd-xsxxd"]
Jul 29 04:29:08.427716 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:29:08.427684    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-a77d4166-6bd7bb48cd-xsxxd" podUID="c5223910-1215-481d-b5a0-0e36169a8be3" containerName="payload-processing" containerID="cri-o://3ebd8b5e3b8ca06ffd4338251f5aae8285f4c3b885fd0cff836002819f69ff80" gracePeriod=30
Jul 29 04:29:08.436843 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:29:08.436803    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a77d4166-5c56bc9676-5cdqk"]
Jul 29 04:29:08.437062 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:29:08.437008    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-a77d4166-5c56bc9676-5cdqk" podUID="cdd4d96c-9cba-4b7e-b846-b796378a567a" containerName="payload-pre-processing" containerID="cri-o://492ecdf505d9b52ec4eaea898df6f18bb285729b2b87bc7a558f0ac5f327b533" gracePeriod=30
Jul 29 04:29:08.613567 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:29:08.613333    2570 generic.go:356] "Generic (PLEG): container finished" podID="c5223910-1215-481d-b5a0-0e36169a8be3" containerID="3ebd8b5e3b8ca06ffd4338251f5aae8285f4c3b885fd0cff836002819f69ff80" exitCode=0
Jul 29 04:29:08.613567 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:29:08.613495    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-a77d4166-6bd7bb48cd-xsxxd" event={"ID":"c5223910-1215-481d-b5a0-0e36169a8be3","Type":"ContainerDied","Data":"3ebd8b5e3b8ca06ffd4338251f5aae8285f4c3b885fd0cff836002819f69ff80"}
Jul 29 04:29:08.615789 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:29:08.615763    2570 generic.go:356] "Generic (PLEG): container finished" podID="ef50925c-5b65-4cd1-bf28-8334af42a7c4" containerID="75ac7c849643f24fec392e2a46e8836db3662dbde969a2f8138ad3f2a6cc0b6b" exitCode=0
Jul 29 04:29:08.615950 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:29:08.615910    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a77d4166-b47d659bb-55qg5" event={"ID":"ef50925c-5b65-4cd1-bf28-8334af42a7c4","Type":"ContainerDied","Data":"75ac7c849643f24fec392e2a46e8836db3662dbde969a2f8138ad3f2a6cc0b6b"}
Jul 29 04:29:08.617905 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:29:08.617881    2570 generic.go:356] "Generic (PLEG): container finished" podID="cdd4d96c-9cba-4b7e-b846-b796378a567a" containerID="492ecdf505d9b52ec4eaea898df6f18bb285729b2b87bc7a558f0ac5f327b533" exitCode=0
Jul 29 04:29:08.617993 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:29:08.617932    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a77d4166-5c56bc9676-5cdqk" event={"ID":"cdd4d96c-9cba-4b7e-b846-b796378a567a","Type":"ContainerDied","Data":"492ecdf505d9b52ec4eaea898df6f18bb285729b2b87bc7a558f0ac5f327b533"}
Jul 29 04:29:08.702766 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:29:08.696383    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a77d4166-b47d659bb-55qg5"
Jul 29 04:29:08.734592 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:29:08.734564    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-a77d4166-6bd7bb48cd-xsxxd"
Jul 29 04:29:08.738489 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:29:08.738470    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a77d4166-5c56bc9676-5cdqk"
Jul 29 04:29:08.808504 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:29:08.808475    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7wsqj\" (UniqueName: \"kubernetes.io/projected/cdd4d96c-9cba-4b7e-b846-b796378a567a-kube-api-access-7wsqj\") pod \"cdd4d96c-9cba-4b7e-b846-b796378a567a\" (UID: \"cdd4d96c-9cba-4b7e-b846-b796378a567a\") "
Jul 29 04:29:08.808692 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:29:08.808522    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef50925c-5b65-4cd1-bf28-8334af42a7c4-maas-api-tls\") pod \"ef50925c-5b65-4cd1-bf28-8334af42a7c4\" (UID: \"ef50925c-5b65-4cd1-bf28-8334af42a7c4\") "
Jul 29 04:29:08.808692 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:29:08.808549    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xkfzb\" (UniqueName: \"kubernetes.io/projected/ef50925c-5b65-4cd1-bf28-8334af42a7c4-kube-api-access-xkfzb\") pod \"ef50925c-5b65-4cd1-bf28-8334af42a7c4\" (UID: \"ef50925c-5b65-4cd1-bf28-8334af42a7c4\") "
Jul 29 04:29:08.808692 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:29:08.808657    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef50925c-5b65-4cd1-bf28-8334af42a7c4-openshift-service-ca\") pod \"ef50925c-5b65-4cd1-bf28-8334af42a7c4\" (UID: \"ef50925c-5b65-4cd1-bf28-8334af42a7c4\") "
Jul 29 04:29:08.808692 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:29:08.808682    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cvxbd\" (UniqueName: \"kubernetes.io/projected/c5223910-1215-481d-b5a0-0e36169a8be3-kube-api-access-cvxbd\") pod \"c5223910-1215-481d-b5a0-0e36169a8be3\" (UID: \"c5223910-1215-481d-b5a0-0e36169a8be3\") "
Jul 29 04:29:08.808958 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:29:08.808738    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdd4d96c-9cba-4b7e-b846-b796378a567a-plugins-config-volume\") pod \"cdd4d96c-9cba-4b7e-b846-b796378a567a\" (UID: \"cdd4d96c-9cba-4b7e-b846-b796378a567a\") "
Jul 29 04:29:08.808958 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:29:08.808795    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5223910-1215-481d-b5a0-0e36169a8be3-plugins-config-volume\") pod \"c5223910-1215-481d-b5a0-0e36169a8be3\" (UID: \"c5223910-1215-481d-b5a0-0e36169a8be3\") "
Jul 29 04:29:08.809488 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:29:08.809460    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5223910-1215-481d-b5a0-0e36169a8be3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c5223910-1215-481d-b5a0-0e36169a8be3" (UID: "c5223910-1215-481d-b5a0-0e36169a8be3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 04:29:08.809617 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:29:08.809571    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef50925c-5b65-4cd1-bf28-8334af42a7c4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ef50925c-5b65-4cd1-bf28-8334af42a7c4" (UID: "ef50925c-5b65-4cd1-bf28-8334af42a7c4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 04:29:08.809965 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:29:08.809938    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdd4d96c-9cba-4b7e-b846-b796378a567a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cdd4d96c-9cba-4b7e-b846-b796378a567a" (UID: "cdd4d96c-9cba-4b7e-b846-b796378a567a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 04:29:08.811083 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:29:08.811050    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef50925c-5b65-4cd1-bf28-8334af42a7c4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ef50925c-5b65-4cd1-bf28-8334af42a7c4" (UID: "ef50925c-5b65-4cd1-bf28-8334af42a7c4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 04:29:08.811169 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:29:08.811146    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdd4d96c-9cba-4b7e-b846-b796378a567a-kube-api-access-7wsqj" (OuterVolumeSpecName: "kube-api-access-7wsqj") pod "cdd4d96c-9cba-4b7e-b846-b796378a567a" (UID: "cdd4d96c-9cba-4b7e-b846-b796378a567a"). InnerVolumeSpecName "kube-api-access-7wsqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 04:29:08.811271 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:29:08.811248    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef50925c-5b65-4cd1-bf28-8334af42a7c4-kube-api-access-xkfzb" (OuterVolumeSpecName: "kube-api-access-xkfzb") pod "ef50925c-5b65-4cd1-bf28-8334af42a7c4" (UID: "ef50925c-5b65-4cd1-bf28-8334af42a7c4"). InnerVolumeSpecName "kube-api-access-xkfzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 04:29:08.811749 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:29:08.811732    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5223910-1215-481d-b5a0-0e36169a8be3-kube-api-access-cvxbd" (OuterVolumeSpecName: "kube-api-access-cvxbd") pod "c5223910-1215-481d-b5a0-0e36169a8be3" (UID: "c5223910-1215-481d-b5a0-0e36169a8be3"). InnerVolumeSpecName "kube-api-access-cvxbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 04:29:08.910671 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:29:08.910630    2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5223910-1215-481d-b5a0-0e36169a8be3-plugins-config-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:29:08.910671 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:29:08.910669    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wsqj\" (UniqueName: \"kubernetes.io/projected/cdd4d96c-9cba-4b7e-b846-b796378a567a-kube-api-access-7wsqj\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:29:08.910921 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:29:08.910684    2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef50925c-5b65-4cd1-bf28-8334af42a7c4-maas-api-tls\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:29:08.910921 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:29:08.910701    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkfzb\" (UniqueName: \"kubernetes.io/projected/ef50925c-5b65-4cd1-bf28-8334af42a7c4-kube-api-access-xkfzb\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:29:08.910921 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:29:08.910715    2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef50925c-5b65-4cd1-bf28-8334af42a7c4-openshift-service-ca\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:29:08.910921 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:29:08.910728    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvxbd\" (UniqueName: \"kubernetes.io/projected/c5223910-1215-481d-b5a0-0e36169a8be3-kube-api-access-cvxbd\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:29:08.910921 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:29:08.910742    2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdd4d96c-9cba-4b7e-b846-b796378a567a-plugins-config-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:29:09.624538 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:29:09.624510    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a77d4166-b47d659bb-55qg5"
Jul 29 04:29:09.624538 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:29:09.624527    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a77d4166-b47d659bb-55qg5" event={"ID":"ef50925c-5b65-4cd1-bf28-8334af42a7c4","Type":"ContainerDied","Data":"6579ffebd9ee9db1f97266f7456e72c196fec56e6cb8afb4e8e691e1261abae6"}
Jul 29 04:29:09.625013 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:29:09.624587    2570 scope.go:117] "RemoveContainer" containerID="75ac7c849643f24fec392e2a46e8836db3662dbde969a2f8138ad3f2a6cc0b6b"
Jul 29 04:29:09.625966 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:29:09.625948    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a77d4166-5c56bc9676-5cdqk"
Jul 29 04:29:09.626047 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:29:09.625965    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a77d4166-5c56bc9676-5cdqk" event={"ID":"cdd4d96c-9cba-4b7e-b846-b796378a567a","Type":"ContainerDied","Data":"e30c50c0aac81c6d914665452d38cc9fd43e66a7b8a1b26f4438f87309029c69"}
Jul 29 04:29:09.627332 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:29:09.627308    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-a77d4166-6bd7bb48cd-xsxxd" event={"ID":"c5223910-1215-481d-b5a0-0e36169a8be3","Type":"ContainerDied","Data":"03fbf76128c2902f8be34a6fafe15cb4730667f518937caecd04f432862843a1"}
Jul 29 04:29:09.627460 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:29:09.627320    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-a77d4166-6bd7bb48cd-xsxxd"
Jul 29 04:29:09.634842 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:29:09.634788    2570 scope.go:117] "RemoveContainer" containerID="492ecdf505d9b52ec4eaea898df6f18bb285729b2b87bc7a558f0ac5f327b533"
Jul 29 04:29:09.643044 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:29:09.643030    2570 scope.go:117] "RemoveContainer" containerID="3ebd8b5e3b8ca06ffd4338251f5aae8285f4c3b885fd0cff836002819f69ff80"
Jul 29 04:29:09.653695 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:29:09.653674    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a77d4166-b47d659bb-55qg5"]
Jul 29 04:29:09.657426 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:29:09.657406    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a77d4166-b47d659bb-55qg5"]
Jul 29 04:29:09.669724 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:29:09.669702    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a77d4166-6bd7bb48cd-xsxxd"]
Jul 29 04:29:09.679248 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:29:09.679226    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a77d4166-6bd7bb48cd-xsxxd"]
Jul 29 04:29:09.690890 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:29:09.690868    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a77d4166-5c56bc9676-5cdqk"]
Jul 29 04:29:09.694876 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:29:09.694858    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a77d4166-5c56bc9676-5cdqk"]
Jul 29 04:29:09.835433 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:29:09.835401    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5223910-1215-481d-b5a0-0e36169a8be3" path="/var/lib/kubelet/pods/c5223910-1215-481d-b5a0-0e36169a8be3/volumes"
Jul 29 04:29:09.835768 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:29:09.835755    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cdd4d96c-9cba-4b7e-b846-b796378a567a" path="/var/lib/kubelet/pods/cdd4d96c-9cba-4b7e-b846-b796378a567a/volumes"
Jul 29 04:29:09.836139 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:29:09.836125    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef50925c-5b65-4cd1-bf28-8334af42a7c4" path="/var/lib/kubelet/pods/ef50925c-5b65-4cd1-bf28-8334af42a7c4/volumes"
Jul 29 04:33:12.877522 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:12.877434    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bd62a50a-86db79cc87-tz89f"]
Jul 29 04:33:12.880036 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:12.877950    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdd4d96c-9cba-4b7e-b846-b796378a567a" containerName="payload-pre-processing"
Jul 29 04:33:12.880036 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:12.877967    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdd4d96c-9cba-4b7e-b846-b796378a567a" containerName="payload-pre-processing"
Jul 29 04:33:12.880036 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:12.877979    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5223910-1215-481d-b5a0-0e36169a8be3" containerName="payload-processing"
Jul 29 04:33:12.880036 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:12.877986    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5223910-1215-481d-b5a0-0e36169a8be3" containerName="payload-processing"
Jul 29 04:33:12.880036 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:12.878000    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef50925c-5b65-4cd1-bf28-8334af42a7c4" containerName="maas-api"
Jul 29 04:33:12.880036 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:12.878005    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef50925c-5b65-4cd1-bf28-8334af42a7c4" containerName="maas-api"
Jul 29 04:33:12.880036 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:12.878077    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef50925c-5b65-4cd1-bf28-8334af42a7c4" containerName="maas-api"
Jul 29 04:33:12.880036 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:12.878086    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdd4d96c-9cba-4b7e-b846-b796378a567a" containerName="payload-pre-processing"
Jul 29 04:33:12.880036 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:12.878093    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5223910-1215-481d-b5a0-0e36169a8be3" containerName="payload-processing"
Jul 29 04:33:12.881006 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:12.880988    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd62a50a-86db79cc87-tz89f"
Jul 29 04:33:12.883872 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:12.883850    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-bd62a50a-serving-cert\""
Jul 29 04:33:12.884342 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:12.884320    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 29 04:33:12.884438 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:12.884319    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 29 04:33:12.885603 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:12.885583    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-lh6p9\""
Jul 29 04:33:12.891953 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:12.891928    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bd62a50a-86db79cc87-tz89f"]
Jul 29 04:33:13.032786 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:13.032743    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shpzx\" (UniqueName: \"kubernetes.io/projected/efbea725-8c24-4ec1-a591-ed159c827b2a-kube-api-access-shpzx\") pod \"maas-api-e2e-mt-bd62a50a-86db79cc87-tz89f\" (UID: \"efbea725-8c24-4ec1-a591-ed159c827b2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd62a50a-86db79cc87-tz89f"
Jul 29 04:33:13.032786 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:13.032791    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/efbea725-8c24-4ec1-a591-ed159c827b2a-openshift-service-ca\") pod \"maas-api-e2e-mt-bd62a50a-86db79cc87-tz89f\" (UID: \"efbea725-8c24-4ec1-a591-ed159c827b2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd62a50a-86db79cc87-tz89f"
Jul 29 04:33:13.033024 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:13.032879    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efbea725-8c24-4ec1-a591-ed159c827b2a-maas-api-tls\") pod \"maas-api-e2e-mt-bd62a50a-86db79cc87-tz89f\" (UID: \"efbea725-8c24-4ec1-a591-ed159c827b2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd62a50a-86db79cc87-tz89f"
Jul 29 04:33:13.134400 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:13.134320    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efbea725-8c24-4ec1-a591-ed159c827b2a-maas-api-tls\") pod \"maas-api-e2e-mt-bd62a50a-86db79cc87-tz89f\" (UID: \"efbea725-8c24-4ec1-a591-ed159c827b2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd62a50a-86db79cc87-tz89f"
Jul 29 04:33:13.134546 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:13.134421    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shpzx\" (UniqueName: \"kubernetes.io/projected/efbea725-8c24-4ec1-a591-ed159c827b2a-kube-api-access-shpzx\") pod \"maas-api-e2e-mt-bd62a50a-86db79cc87-tz89f\" (UID: \"efbea725-8c24-4ec1-a591-ed159c827b2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd62a50a-86db79cc87-tz89f"
Jul 29 04:33:13.134546 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:13.134457    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/efbea725-8c24-4ec1-a591-ed159c827b2a-openshift-service-ca\") pod \"maas-api-e2e-mt-bd62a50a-86db79cc87-tz89f\" (UID: \"efbea725-8c24-4ec1-a591-ed159c827b2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd62a50a-86db79cc87-tz89f"
Jul 29 04:33:13.135144 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:13.135124    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/efbea725-8c24-4ec1-a591-ed159c827b2a-openshift-service-ca\") pod \"maas-api-e2e-mt-bd62a50a-86db79cc87-tz89f\" (UID: \"efbea725-8c24-4ec1-a591-ed159c827b2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd62a50a-86db79cc87-tz89f"
Jul 29 04:33:13.136931 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:13.136913    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efbea725-8c24-4ec1-a591-ed159c827b2a-maas-api-tls\") pod \"maas-api-e2e-mt-bd62a50a-86db79cc87-tz89f\" (UID: \"efbea725-8c24-4ec1-a591-ed159c827b2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd62a50a-86db79cc87-tz89f"
Jul 29 04:33:13.146186 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:13.146167    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shpzx\" (UniqueName: \"kubernetes.io/projected/efbea725-8c24-4ec1-a591-ed159c827b2a-kube-api-access-shpzx\") pod \"maas-api-e2e-mt-bd62a50a-86db79cc87-tz89f\" (UID: \"efbea725-8c24-4ec1-a591-ed159c827b2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd62a50a-86db79cc87-tz89f"
Jul 29 04:33:13.191505 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:13.191480    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd62a50a-86db79cc87-tz89f"
Jul 29 04:33:13.333809 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:13.333788    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bd62a50a-86db79cc87-tz89f"]
Jul 29 04:33:13.335400 ip-10-0-135-138 kubenswrapper[2570]: W0729 04:33:13.335366    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefbea725_8c24_4ec1_a591_ed159c827b2a.slice/crio-45311b7db91d66211f3a03abb7e90f81f39defa91a1825823558415990fcb94e WatchSource:0}: Error finding container 45311b7db91d66211f3a03abb7e90f81f39defa91a1825823558415990fcb94e: Status 404 returned error can't find the container with id 45311b7db91d66211f3a03abb7e90f81f39defa91a1825823558415990fcb94e
Jul 29 04:33:13.337118 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:13.337099    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 04:33:13.713138 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:13.713105    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bd62a50a-7d944bcd7b-qc7v6"]
Jul 29 04:33:13.716731 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:13.716703    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-bd62a50a-7d944bcd7b-qc7v6"
Jul 29 04:33:13.719808 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:13.719784    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-bd62a50a\""
Jul 29 04:33:13.726940 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:13.726919    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bd62a50a-7d944bcd7b-qc7v6"]
Jul 29 04:33:13.730868 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:13.730845    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-bd62a50a-ff56558cd-x2xln"]
Jul 29 04:33:13.736547 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:13.736527    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-bd62a50a-ff56558cd-x2xln"
Jul 29 04:33:13.743205 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:13.743157    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-bd62a50a-dockercfg-crl2g\""
Jul 29 04:33:13.752672 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:13.752648    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-bd62a50a-ff56558cd-x2xln"]
Jul 29 04:33:13.813980 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:13.813950    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd62a50a-86db79cc87-tz89f" event={"ID":"efbea725-8c24-4ec1-a591-ed159c827b2a","Type":"ContainerStarted","Data":"3669dce236fc6556dfa1bed22540278cda3422944a1fd78c5e0cd6a297aa0b2a"}
Jul 29 04:33:13.814133 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:13.814010    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd62a50a-86db79cc87-tz89f" event={"ID":"efbea725-8c24-4ec1-a591-ed159c827b2a","Type":"ContainerStarted","Data":"45311b7db91d66211f3a03abb7e90f81f39defa91a1825823558415990fcb94e"}
Jul 29 04:33:13.814133 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:13.814054    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd62a50a-86db79cc87-tz89f"
Jul 29 04:33:13.840034 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:13.840006    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf242\" (UniqueName: \"kubernetes.io/projected/57f84873-2bef-42fb-b337-e2da4bd475e5-kube-api-access-qf242\") pod \"payload-pre-processing-e2e-mt-bd62a50a-7d944bcd7b-qc7v6\" (UID: \"57f84873-2bef-42fb-b337-e2da4bd475e5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bd62a50a-7d944bcd7b-qc7v6"
Jul 29 04:33:13.840160 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:13.840121    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4914f52b-c0da-4f3a-806d-d677e912c519-plugins-config-volume\") pod \"payload-processing-e2e-mt-bd62a50a-ff56558cd-x2xln\" (UID: \"4914f52b-c0da-4f3a-806d-d677e912c519\") " pod="openshift-ingress/payload-processing-e2e-mt-bd62a50a-ff56558cd-x2xln"
Jul 29 04:33:13.840313 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:13.840296    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57f84873-2bef-42fb-b337-e2da4bd475e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-bd62a50a-7d944bcd7b-qc7v6\" (UID: \"57f84873-2bef-42fb-b337-e2da4bd475e5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bd62a50a-7d944bcd7b-qc7v6"
Jul 29 04:33:13.840387 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:13.840321    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhhj8\" (UniqueName: \"kubernetes.io/projected/4914f52b-c0da-4f3a-806d-d677e912c519-kube-api-access-lhhj8\") pod \"payload-processing-e2e-mt-bd62a50a-ff56558cd-x2xln\" (UID: \"4914f52b-c0da-4f3a-806d-d677e912c519\") " pod="openshift-ingress/payload-processing-e2e-mt-bd62a50a-ff56558cd-x2xln"
Jul 29 04:33:13.854134 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:13.854089    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd62a50a-86db79cc87-tz89f" podStartSLOduration=1.452928717 podStartE2EDuration="1.854076595s" totalImagesPullingTime="401.147878ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:33:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:33:13.853650005 +0000 UTC m=+2258.583781095" watchObservedRunningTime="2026-07-29 04:33:13.854076595 +0000 UTC m=+2258.584207674"
Jul 29 04:33:13.941715 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:13.941675    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57f84873-2bef-42fb-b337-e2da4bd475e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-bd62a50a-7d944bcd7b-qc7v6\" (UID: \"57f84873-2bef-42fb-b337-e2da4bd475e5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bd62a50a-7d944bcd7b-qc7v6"
Jul 29 04:33:13.941715 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:13.941713    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhhj8\" (UniqueName: \"kubernetes.io/projected/4914f52b-c0da-4f3a-806d-d677e912c519-kube-api-access-lhhj8\") pod \"payload-processing-e2e-mt-bd62a50a-ff56558cd-x2xln\" (UID: \"4914f52b-c0da-4f3a-806d-d677e912c519\") " pod="openshift-ingress/payload-processing-e2e-mt-bd62a50a-ff56558cd-x2xln"
Jul 29 04:33:13.942242 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:13.941804    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf242\" (UniqueName: \"kubernetes.io/projected/57f84873-2bef-42fb-b337-e2da4bd475e5-kube-api-access-qf242\") pod \"payload-pre-processing-e2e-mt-bd62a50a-7d944bcd7b-qc7v6\" (UID: \"57f84873-2bef-42fb-b337-e2da4bd475e5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bd62a50a-7d944bcd7b-qc7v6"
Jul 29 04:33:13.942242 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:13.941899    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4914f52b-c0da-4f3a-806d-d677e912c519-plugins-config-volume\") pod \"payload-processing-e2e-mt-bd62a50a-ff56558cd-x2xln\" (UID: \"4914f52b-c0da-4f3a-806d-d677e912c519\") " pod="openshift-ingress/payload-processing-e2e-mt-bd62a50a-ff56558cd-x2xln"
Jul 29 04:33:13.942502 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:13.942480    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57f84873-2bef-42fb-b337-e2da4bd475e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-bd62a50a-7d944bcd7b-qc7v6\" (UID: \"57f84873-2bef-42fb-b337-e2da4bd475e5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bd62a50a-7d944bcd7b-qc7v6"
Jul 29 04:33:13.942694 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:13.942674    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4914f52b-c0da-4f3a-806d-d677e912c519-plugins-config-volume\") pod \"payload-processing-e2e-mt-bd62a50a-ff56558cd-x2xln\" (UID: \"4914f52b-c0da-4f3a-806d-d677e912c519\") " pod="openshift-ingress/payload-processing-e2e-mt-bd62a50a-ff56558cd-x2xln"
Jul 29 04:33:13.950836 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:13.950798    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf242\" (UniqueName: \"kubernetes.io/projected/57f84873-2bef-42fb-b337-e2da4bd475e5-kube-api-access-qf242\") pod \"payload-pre-processing-e2e-mt-bd62a50a-7d944bcd7b-qc7v6\" (UID: \"57f84873-2bef-42fb-b337-e2da4bd475e5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bd62a50a-7d944bcd7b-qc7v6"
Jul 29 04:33:13.950947 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:13.950877    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhhj8\" (UniqueName: \"kubernetes.io/projected/4914f52b-c0da-4f3a-806d-d677e912c519-kube-api-access-lhhj8\") pod \"payload-processing-e2e-mt-bd62a50a-ff56558cd-x2xln\" (UID: \"4914f52b-c0da-4f3a-806d-d677e912c519\") " pod="openshift-ingress/payload-processing-e2e-mt-bd62a50a-ff56558cd-x2xln"
Jul 29 04:33:14.028983 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:14.028907    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-bd62a50a-7d944bcd7b-qc7v6"
Jul 29 04:33:14.047311 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:14.047282    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-bd62a50a-ff56558cd-x2xln"
Jul 29 04:33:14.186116 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:14.184989    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bd62a50a-7d944bcd7b-qc7v6"]
Jul 29 04:33:14.420630 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:14.420603    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-bd62a50a-ff56558cd-x2xln"]
Jul 29 04:33:14.422686 ip-10-0-135-138 kubenswrapper[2570]: W0729 04:33:14.422652    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4914f52b_c0da_4f3a_806d_d677e912c519.slice/crio-c82622c3b4fe3007eb7b4ca6fbaea1f90785856af97dd2c208a7af16409b09d8 WatchSource:0}: Error finding container c82622c3b4fe3007eb7b4ca6fbaea1f90785856af97dd2c208a7af16409b09d8: Status 404 returned error can't find the container with id c82622c3b4fe3007eb7b4ca6fbaea1f90785856af97dd2c208a7af16409b09d8
Jul 29 04:33:14.821612 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:14.821525    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-bd62a50a-ff56558cd-x2xln" event={"ID":"4914f52b-c0da-4f3a-806d-d677e912c519","Type":"ContainerStarted","Data":"dbdf9535adc61490383cc0a93f8417ff2e0f4e652d79d06dbbec809f33f2602d"}
Jul 29 04:33:14.821612 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:14.821585    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-bd62a50a-ff56558cd-x2xln" event={"ID":"4914f52b-c0da-4f3a-806d-d677e912c519","Type":"ContainerStarted","Data":"c82622c3b4fe3007eb7b4ca6fbaea1f90785856af97dd2c208a7af16409b09d8"}
Jul 29 04:33:14.821878 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:14.821634    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-bd62a50a-ff56558cd-x2xln"
Jul 29 04:33:14.823533 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:14.823496    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-bd62a50a-7d944bcd7b-qc7v6" event={"ID":"57f84873-2bef-42fb-b337-e2da4bd475e5","Type":"ContainerStarted","Data":"e77466b9ea5e4d32e5e57bdfbd8ad8b2061948d3de5bc45357551dfe6cafefca"}
Jul 29 04:33:14.823662 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:14.823544    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-bd62a50a-7d944bcd7b-qc7v6" event={"ID":"57f84873-2bef-42fb-b337-e2da4bd475e5","Type":"ContainerStarted","Data":"6836aac4c7e6505c1bd10f5756bc761cc5e6ba3f7b11d3e6c78191e7ca9f25d4"}
Jul 29 04:33:14.823662 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:14.823650    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-bd62a50a-7d944bcd7b-qc7v6"
Jul 29 04:33:14.839630 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:14.839577    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-bd62a50a-ff56558cd-x2xln" podStartSLOduration=1.839564619 podStartE2EDuration="1.839564619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:33:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:33:14.838925691 +0000 UTC m=+2259.569056807" watchObservedRunningTime="2026-07-29 04:33:14.839564619 +0000 UTC m=+2259.569695689"
Jul 29 04:33:14.875165 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:14.875116    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-bd62a50a-7d944bcd7b-qc7v6" podStartSLOduration=1.875102004 podStartE2EDuration="1.875102004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:33:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:33:14.865641879 +0000 UTC m=+2259.595772969" watchObservedRunningTime="2026-07-29 04:33:14.875102004 +0000 UTC m=+2259.605233114"
Jul 29 04:33:19.834363 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:19.834334    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd62a50a-86db79cc87-tz89f"
Jul 29 04:33:25.834948 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:25.834921    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-bd62a50a-ff56558cd-x2xln"
Jul 29 04:33:25.835336 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:25.834973    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-bd62a50a-7d944bcd7b-qc7v6"
Jul 29 04:33:35.972230 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:35.972201    2570 scope.go:117] "RemoveContainer" containerID="5fa9dc31a7ffaebefbe74f3f871656fc3041a599595524e9121962c85c28d1f0"
Jul 29 04:33:45.250572 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:45.250539    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bd62a50a-q74cp"]
Jul 29 04:33:45.254087 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:45.254067    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bd62a50a-q74cp"
Jul 29 04:33:45.257507 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:45.257485    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8frm\""
Jul 29 04:33:45.264930 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:45.264910    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bd62a50a-q74cp"]
Jul 29 04:33:45.334369 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:45.334343    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/063a7c97-fbf9-4f90-b1b9-30e38338716c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-bd62a50a-q74cp\" (UID: \"063a7c97-fbf9-4f90-b1b9-30e38338716c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bd62a50a-q74cp"
Jul 29 04:33:45.434909 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:45.434876    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/063a7c97-fbf9-4f90-b1b9-30e38338716c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-bd62a50a-q74cp\" (UID: \"063a7c97-fbf9-4f90-b1b9-30e38338716c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bd62a50a-q74cp"
Jul 29 04:33:45.435527 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:45.435501    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/063a7c97-fbf9-4f90-b1b9-30e38338716c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-bd62a50a-q74cp\" (UID: \"063a7c97-fbf9-4f90-b1b9-30e38338716c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bd62a50a-q74cp"
Jul 29 04:33:45.564696 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:45.564628    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bd62a50a-q74cp"
Jul 29 04:33:45.695806 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:45.695783    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bd62a50a-q74cp"]
Jul 29 04:33:45.698568 ip-10-0-135-138 kubenswrapper[2570]: W0729 04:33:45.698539    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod063a7c97_fbf9_4f90_b1b9_30e38338716c.slice/crio-fea3885d86a59fd62006fe3df9e5e3e2c437d7833a28016a5c464a7dc72c849e WatchSource:0}: Error finding container fea3885d86a59fd62006fe3df9e5e3e2c437d7833a28016a5c464a7dc72c849e: Status 404 returned error can't find the container with id fea3885d86a59fd62006fe3df9e5e3e2c437d7833a28016a5c464a7dc72c849e
Jul 29 04:33:45.979935 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:45.979898    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bd62a50a-q74cp" event={"ID":"063a7c97-fbf9-4f90-b1b9-30e38338716c","Type":"ContainerStarted","Data":"fea3885d86a59fd62006fe3df9e5e3e2c437d7833a28016a5c464a7dc72c849e"}
Jul 29 04:33:46.985990 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:46.985954    2570 generic.go:356] "Generic (PLEG): container finished" podID="063a7c97-fbf9-4f90-b1b9-30e38338716c" containerID="6644b716e4429a8c8cffe521f7b89f8d6c3fc635767403b2618e02823615f407" exitCode=0
Jul 29 04:33:46.985990 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:46.985990    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bd62a50a-q74cp" event={"ID":"063a7c97-fbf9-4f90-b1b9-30e38338716c","Type":"ContainerDied","Data":"6644b716e4429a8c8cffe521f7b89f8d6c3fc635767403b2618e02823615f407"}
Jul 29 04:33:48.123224 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:48.123204    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bd62a50a-q74cp"
Jul 29 04:33:48.261119 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:48.261047    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/063a7c97-fbf9-4f90-b1b9-30e38338716c-maas-api-service-ca\") pod \"063a7c97-fbf9-4f90-b1b9-30e38338716c\" (UID: \"063a7c97-fbf9-4f90-b1b9-30e38338716c\") "
Jul 29 04:33:48.261378 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:48.261356    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/063a7c97-fbf9-4f90-b1b9-30e38338716c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "063a7c97-fbf9-4f90-b1b9-30e38338716c" (UID: "063a7c97-fbf9-4f90-b1b9-30e38338716c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 04:33:48.362125 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:48.362099    2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/063a7c97-fbf9-4f90-b1b9-30e38338716c-maas-api-service-ca\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:33:48.996792 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:48.996763    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bd62a50a-q74cp"
Jul 29 04:33:48.996976 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:48.996761    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bd62a50a-q74cp" event={"ID":"063a7c97-fbf9-4f90-b1b9-30e38338716c","Type":"ContainerDied","Data":"fea3885d86a59fd62006fe3df9e5e3e2c437d7833a28016a5c464a7dc72c849e"}
Jul 29 04:33:48.996976 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:48.996894    2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fea3885d86a59fd62006fe3df9e5e3e2c437d7833a28016a5c464a7dc72c849e"
Jul 29 04:33:55.785171 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:55.785139    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bd62a50a-86db79cc87-tz89f"]
Jul 29 04:33:55.785633 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:55.785334    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd62a50a-86db79cc87-tz89f" podUID="efbea725-8c24-4ec1-a591-ed159c827b2a" containerName="maas-api" containerID="cri-o://3669dce236fc6556dfa1bed22540278cda3422944a1fd78c5e0cd6a297aa0b2a" gracePeriod=30
Jul 29 04:33:55.866538 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:55.866505    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-bd62a50a-ff56558cd-x2xln"]
Jul 29 04:33:55.866705 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:55.866684    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-bd62a50a-ff56558cd-x2xln" podUID="4914f52b-c0da-4f3a-806d-d677e912c519" containerName="payload-processing" containerID="cri-o://dbdf9535adc61490383cc0a93f8417ff2e0f4e652d79d06dbbec809f33f2602d" gracePeriod=30
Jul 29 04:33:55.880610 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:55.880583    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bd62a50a-7d944bcd7b-qc7v6"]
Jul 29 04:33:55.880765 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:55.880746    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-bd62a50a-7d944bcd7b-qc7v6" podUID="57f84873-2bef-42fb-b337-e2da4bd475e5" containerName="payload-pre-processing" containerID="cri-o://e77466b9ea5e4d32e5e57bdfbd8ad8b2061948d3de5bc45357551dfe6cafefca" gracePeriod=30
Jul 29 04:33:56.047288 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:56.043807    2570 generic.go:356] "Generic (PLEG): container finished" podID="4914f52b-c0da-4f3a-806d-d677e912c519" containerID="dbdf9535adc61490383cc0a93f8417ff2e0f4e652d79d06dbbec809f33f2602d" exitCode=0
Jul 29 04:33:56.047288 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:56.044124    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-bd62a50a-ff56558cd-x2xln" event={"ID":"4914f52b-c0da-4f3a-806d-d677e912c519","Type":"ContainerDied","Data":"dbdf9535adc61490383cc0a93f8417ff2e0f4e652d79d06dbbec809f33f2602d"}
Jul 29 04:33:56.049655 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:56.049624    2570 generic.go:356] "Generic (PLEG): container finished" podID="57f84873-2bef-42fb-b337-e2da4bd475e5" containerID="e77466b9ea5e4d32e5e57bdfbd8ad8b2061948d3de5bc45357551dfe6cafefca" exitCode=0
Jul 29 04:33:56.049797 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:56.049691    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-bd62a50a-7d944bcd7b-qc7v6" event={"ID":"57f84873-2bef-42fb-b337-e2da4bd475e5","Type":"ContainerDied","Data":"e77466b9ea5e4d32e5e57bdfbd8ad8b2061948d3de5bc45357551dfe6cafefca"}
Jul 29 04:33:56.051580 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:56.051555    2570 generic.go:356] "Generic (PLEG): container finished" podID="efbea725-8c24-4ec1-a591-ed159c827b2a" containerID="3669dce236fc6556dfa1bed22540278cda3422944a1fd78c5e0cd6a297aa0b2a" exitCode=0
Jul 29 04:33:56.051679 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:56.051582    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd62a50a-86db79cc87-tz89f" event={"ID":"efbea725-8c24-4ec1-a591-ed159c827b2a","Type":"ContainerDied","Data":"3669dce236fc6556dfa1bed22540278cda3422944a1fd78c5e0cd6a297aa0b2a"}
Jul 29 04:33:56.071225 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:56.071201    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bd62a50a-q74cp"]
Jul 29 04:33:56.074136 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:56.074114    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bd62a50a-q74cp"]
Jul 29 04:33:56.162462 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:56.162436    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-bd62a50a-ff56558cd-x2xln"
Jul 29 04:33:56.185682 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:56.185657    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd62a50a-86db79cc87-tz89f"
Jul 29 04:33:56.194775 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:56.194754    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-bd62a50a-7d944bcd7b-qc7v6"
Jul 29 04:33:56.328432 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:56.328346    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efbea725-8c24-4ec1-a591-ed159c827b2a-maas-api-tls\") pod \"efbea725-8c24-4ec1-a591-ed159c827b2a\" (UID: \"efbea725-8c24-4ec1-a591-ed159c827b2a\") "
Jul 29 04:33:56.328432 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:56.328422    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-shpzx\" (UniqueName: \"kubernetes.io/projected/efbea725-8c24-4ec1-a591-ed159c827b2a-kube-api-access-shpzx\") pod \"efbea725-8c24-4ec1-a591-ed159c827b2a\" (UID: \"efbea725-8c24-4ec1-a591-ed159c827b2a\") "
Jul 29 04:33:56.328681 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:56.328456    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/efbea725-8c24-4ec1-a591-ed159c827b2a-openshift-service-ca\") pod \"efbea725-8c24-4ec1-a591-ed159c827b2a\" (UID: \"efbea725-8c24-4ec1-a591-ed159c827b2a\") "
Jul 29 04:33:56.328681 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:56.328472    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4914f52b-c0da-4f3a-806d-d677e912c519-plugins-config-volume\") pod \"4914f52b-c0da-4f3a-806d-d677e912c519\" (UID: \"4914f52b-c0da-4f3a-806d-d677e912c519\") "
Jul 29 04:33:56.328681 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:56.328502    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qf242\" (UniqueName: \"kubernetes.io/projected/57f84873-2bef-42fb-b337-e2da4bd475e5-kube-api-access-qf242\") pod \"57f84873-2bef-42fb-b337-e2da4bd475e5\" (UID: \"57f84873-2bef-42fb-b337-e2da4bd475e5\") "
Jul 29 04:33:56.328681 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:56.328564    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57f84873-2bef-42fb-b337-e2da4bd475e5-plugins-config-volume\") pod \"57f84873-2bef-42fb-b337-e2da4bd475e5\" (UID: \"57f84873-2bef-42fb-b337-e2da4bd475e5\") "
Jul 29 04:33:56.328681 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:56.328645    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lhhj8\" (UniqueName: \"kubernetes.io/projected/4914f52b-c0da-4f3a-806d-d677e912c519-kube-api-access-lhhj8\") pod \"4914f52b-c0da-4f3a-806d-d677e912c519\" (UID: \"4914f52b-c0da-4f3a-806d-d677e912c519\") "
Jul 29 04:33:56.328997 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:56.328890    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4914f52b-c0da-4f3a-806d-d677e912c519-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4914f52b-c0da-4f3a-806d-d677e912c519" (UID: "4914f52b-c0da-4f3a-806d-d677e912c519"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 04:33:56.329060 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:56.329040    2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4914f52b-c0da-4f3a-806d-d677e912c519-plugins-config-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:33:56.330707 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:56.330491    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efbea725-8c24-4ec1-a591-ed159c827b2a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "efbea725-8c24-4ec1-a591-ed159c827b2a" (UID: "efbea725-8c24-4ec1-a591-ed159c827b2a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 04:33:56.330707 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:56.330686    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57f84873-2bef-42fb-b337-e2da4bd475e5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "57f84873-2bef-42fb-b337-e2da4bd475e5" (UID: "57f84873-2bef-42fb-b337-e2da4bd475e5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 04:33:56.331249 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:56.331191    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efbea725-8c24-4ec1-a591-ed159c827b2a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "efbea725-8c24-4ec1-a591-ed159c827b2a" (UID: "efbea725-8c24-4ec1-a591-ed159c827b2a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 04:33:56.331403 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:56.331380    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4914f52b-c0da-4f3a-806d-d677e912c519-kube-api-access-lhhj8" (OuterVolumeSpecName: "kube-api-access-lhhj8") pod "4914f52b-c0da-4f3a-806d-d677e912c519" (UID: "4914f52b-c0da-4f3a-806d-d677e912c519"). InnerVolumeSpecName "kube-api-access-lhhj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 04:33:56.331510 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:56.331396    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57f84873-2bef-42fb-b337-e2da4bd475e5-kube-api-access-qf242" (OuterVolumeSpecName: "kube-api-access-qf242") pod "57f84873-2bef-42fb-b337-e2da4bd475e5" (UID: "57f84873-2bef-42fb-b337-e2da4bd475e5"). InnerVolumeSpecName "kube-api-access-qf242". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 04:33:56.331510 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:56.331464    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efbea725-8c24-4ec1-a591-ed159c827b2a-kube-api-access-shpzx" (OuterVolumeSpecName: "kube-api-access-shpzx") pod "efbea725-8c24-4ec1-a591-ed159c827b2a" (UID: "efbea725-8c24-4ec1-a591-ed159c827b2a"). InnerVolumeSpecName "kube-api-access-shpzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 04:33:56.430204 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:56.430171    2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57f84873-2bef-42fb-b337-e2da4bd475e5-plugins-config-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:33:56.430204 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:56.430201    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhhj8\" (UniqueName: \"kubernetes.io/projected/4914f52b-c0da-4f3a-806d-d677e912c519-kube-api-access-lhhj8\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:33:56.430398 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:56.430214    2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efbea725-8c24-4ec1-a591-ed159c827b2a-maas-api-tls\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:33:56.430398 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:56.430227    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shpzx\" (UniqueName: \"kubernetes.io/projected/efbea725-8c24-4ec1-a591-ed159c827b2a-kube-api-access-shpzx\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:33:56.430398 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:56.430238    2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/efbea725-8c24-4ec1-a591-ed159c827b2a-openshift-service-ca\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:33:56.430398 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:56.430250    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qf242\" (UniqueName: \"kubernetes.io/projected/57f84873-2bef-42fb-b337-e2da4bd475e5-kube-api-access-qf242\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:33:57.058334 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:57.058245    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd62a50a-86db79cc87-tz89f"
Jul 29 04:33:57.058334 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:57.058276    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd62a50a-86db79cc87-tz89f" event={"ID":"efbea725-8c24-4ec1-a591-ed159c827b2a","Type":"ContainerDied","Data":"45311b7db91d66211f3a03abb7e90f81f39defa91a1825823558415990fcb94e"}
Jul 29 04:33:57.058895 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:57.058341    2570 scope.go:117] "RemoveContainer" containerID="3669dce236fc6556dfa1bed22540278cda3422944a1fd78c5e0cd6a297aa0b2a"
Jul 29 04:33:57.059786 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:57.059771    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-bd62a50a-ff56558cd-x2xln"
Jul 29 04:33:57.059982 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:57.059781    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-bd62a50a-ff56558cd-x2xln" event={"ID":"4914f52b-c0da-4f3a-806d-d677e912c519","Type":"ContainerDied","Data":"c82622c3b4fe3007eb7b4ca6fbaea1f90785856af97dd2c208a7af16409b09d8"}
Jul 29 04:33:57.061212 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:57.061190    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-bd62a50a-7d944bcd7b-qc7v6" event={"ID":"57f84873-2bef-42fb-b337-e2da4bd475e5","Type":"ContainerDied","Data":"6836aac4c7e6505c1bd10f5756bc761cc5e6ba3f7b11d3e6c78191e7ca9f25d4"}
Jul 29 04:33:57.061322 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:57.061220    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-bd62a50a-7d944bcd7b-qc7v6"
Jul 29 04:33:57.068524 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:57.068500    2570 scope.go:117] "RemoveContainer" containerID="dbdf9535adc61490383cc0a93f8417ff2e0f4e652d79d06dbbec809f33f2602d"
Jul 29 04:33:57.077413 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:57.077397    2570 scope.go:117] "RemoveContainer" containerID="e77466b9ea5e4d32e5e57bdfbd8ad8b2061948d3de5bc45357551dfe6cafefca"
Jul 29 04:33:57.086439 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:57.086420    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bd62a50a-86db79cc87-tz89f"]
Jul 29 04:33:57.090526 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:57.090508    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bd62a50a-86db79cc87-tz89f"]
Jul 29 04:33:57.102339 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:57.102314    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bd62a50a-7d944bcd7b-qc7v6"]
Jul 29 04:33:57.108466 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:57.108449    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bd62a50a-7d944bcd7b-qc7v6"]
Jul 29 04:33:57.123448 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:57.123427    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-bd62a50a-ff56558cd-x2xln"]
Jul 29 04:33:57.127009 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:57.126985    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-bd62a50a-ff56558cd-x2xln"]
Jul 29 04:33:57.836203 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:57.836172    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="063a7c97-fbf9-4f90-b1b9-30e38338716c" path="/var/lib/kubelet/pods/063a7c97-fbf9-4f90-b1b9-30e38338716c/volumes"
Jul 29 04:33:57.836610 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:57.836592    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4914f52b-c0da-4f3a-806d-d677e912c519" path="/var/lib/kubelet/pods/4914f52b-c0da-4f3a-806d-d677e912c519/volumes"
Jul 29 04:33:57.836992 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:57.836975    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57f84873-2bef-42fb-b337-e2da4bd475e5" path="/var/lib/kubelet/pods/57f84873-2bef-42fb-b337-e2da4bd475e5/volumes"
Jul 29 04:33:57.837375 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:33:57.837359    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="efbea725-8c24-4ec1-a591-ed159c827b2a" path="/var/lib/kubelet/pods/efbea725-8c24-4ec1-a591-ed159c827b2a/volumes"
Jul 29 04:34:43.341320 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:34:43.341283    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-chq2s"]
Jul 29 04:34:43.341815 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:34:43.341502    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-chq2s" podUID="9b6e291d-265a-4aa5-a74f-5763ee7294ae" containerName="manager" containerID="cri-o://cd81f34cbcf921d22645d6b586df7a16a23edf7cb0dec8e891f074a599733036" gracePeriod=10
Jul 29 04:34:44.091156 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:34:44.091129    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-chq2s"
Jul 29 04:34:44.250597 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:34:44.250522    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7c526\" (UniqueName: \"kubernetes.io/projected/9b6e291d-265a-4aa5-a74f-5763ee7294ae-kube-api-access-7c526\") pod \"9b6e291d-265a-4aa5-a74f-5763ee7294ae\" (UID: \"9b6e291d-265a-4aa5-a74f-5763ee7294ae\") "
Jul 29 04:34:44.250597 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:34:44.250563    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9b6e291d-265a-4aa5-a74f-5763ee7294ae-extensions-socket-volume\") pod \"9b6e291d-265a-4aa5-a74f-5763ee7294ae\" (UID: \"9b6e291d-265a-4aa5-a74f-5763ee7294ae\") "
Jul 29 04:34:44.251023 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:34:44.251001    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b6e291d-265a-4aa5-a74f-5763ee7294ae-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "9b6e291d-265a-4aa5-a74f-5763ee7294ae" (UID: "9b6e291d-265a-4aa5-a74f-5763ee7294ae"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 04:34:44.252569 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:34:44.252546    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b6e291d-265a-4aa5-a74f-5763ee7294ae-kube-api-access-7c526" (OuterVolumeSpecName: "kube-api-access-7c526") pod "9b6e291d-265a-4aa5-a74f-5763ee7294ae" (UID: "9b6e291d-265a-4aa5-a74f-5763ee7294ae"). InnerVolumeSpecName "kube-api-access-7c526". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 04:34:44.300892 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:34:44.300855    2570 generic.go:356] "Generic (PLEG): container finished" podID="9b6e291d-265a-4aa5-a74f-5763ee7294ae" containerID="cd81f34cbcf921d22645d6b586df7a16a23edf7cb0dec8e891f074a599733036" exitCode=0
Jul 29 04:34:44.301023 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:34:44.300954    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-chq2s"
Jul 29 04:34:44.301023 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:34:44.300974    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-chq2s" event={"ID":"9b6e291d-265a-4aa5-a74f-5763ee7294ae","Type":"ContainerDied","Data":"cd81f34cbcf921d22645d6b586df7a16a23edf7cb0dec8e891f074a599733036"}
Jul 29 04:34:44.301139 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:34:44.301030    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-chq2s" event={"ID":"9b6e291d-265a-4aa5-a74f-5763ee7294ae","Type":"ContainerDied","Data":"4aaf293d03a3f5abcbfd5a8a82e61ca278f5359505412c358642521981024517"}
Jul 29 04:34:44.301139 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:34:44.301062    2570 scope.go:117] "RemoveContainer" containerID="cd81f34cbcf921d22645d6b586df7a16a23edf7cb0dec8e891f074a599733036"
Jul 29 04:34:44.310895 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:34:44.310876    2570 scope.go:117] "RemoveContainer" containerID="cd81f34cbcf921d22645d6b586df7a16a23edf7cb0dec8e891f074a599733036"
Jul 29 04:34:44.311129 ip-10-0-135-138 kubenswrapper[2570]: E0729 04:34:44.311108    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd81f34cbcf921d22645d6b586df7a16a23edf7cb0dec8e891f074a599733036\": container with ID starting with cd81f34cbcf921d22645d6b586df7a16a23edf7cb0dec8e891f074a599733036 not found: ID does not exist" containerID="cd81f34cbcf921d22645d6b586df7a16a23edf7cb0dec8e891f074a599733036"
Jul 29 04:34:44.311203 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:34:44.311141    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd81f34cbcf921d22645d6b586df7a16a23edf7cb0dec8e891f074a599733036"} err="failed to get container status \"cd81f34cbcf921d22645d6b586df7a16a23edf7cb0dec8e891f074a599733036\": rpc error: code = NotFound desc = could not find container \"cd81f34cbcf921d22645d6b586df7a16a23edf7cb0dec8e891f074a599733036\": container with ID starting with cd81f34cbcf921d22645d6b586df7a16a23edf7cb0dec8e891f074a599733036 not found: ID does not exist"
Jul 29 04:34:44.323575 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:34:44.323545    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-chq2s"]
Jul 29 04:34:44.328430 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:34:44.328411    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-chq2s"]
Jul 29 04:34:44.352006 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:34:44.351982    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7c526\" (UniqueName: \"kubernetes.io/projected/9b6e291d-265a-4aa5-a74f-5763ee7294ae-kube-api-access-7c526\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:34:44.352006 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:34:44.352002    2570 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9b6e291d-265a-4aa5-a74f-5763ee7294ae-extensions-socket-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:34:45.834893 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:34:45.834854    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b6e291d-265a-4aa5-a74f-5763ee7294ae" path="/var/lib/kubelet/pods/9b6e291d-265a-4aa5-a74f-5763ee7294ae/volumes"
Jul 29 04:35:58.957296 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:35:58.957211    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wqt5k"]
Jul 29 04:35:58.957891 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:35:58.957694    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4914f52b-c0da-4f3a-806d-d677e912c519" containerName="payload-processing"
Jul 29 04:35:58.957891 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:35:58.957715    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="4914f52b-c0da-4f3a-806d-d677e912c519" containerName="payload-processing"
Jul 29 04:35:58.957891 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:35:58.957726    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b6e291d-265a-4aa5-a74f-5763ee7294ae" containerName="manager"
Jul 29 04:35:58.957891 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:35:58.957735    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b6e291d-265a-4aa5-a74f-5763ee7294ae" containerName="manager"
Jul 29 04:35:58.957891 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:35:58.957770    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="063a7c97-fbf9-4f90-b1b9-30e38338716c" containerName="revoke-keys"
Jul 29 04:35:58.957891 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:35:58.957779    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="063a7c97-fbf9-4f90-b1b9-30e38338716c" containerName="revoke-keys"
Jul 29 04:35:58.957891 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:35:58.957800    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efbea725-8c24-4ec1-a591-ed159c827b2a" containerName="maas-api"
Jul 29 04:35:58.957891 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:35:58.957807    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="efbea725-8c24-4ec1-a591-ed159c827b2a" containerName="maas-api"
Jul 29 04:35:58.957891 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:35:58.957833    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57f84873-2bef-42fb-b337-e2da4bd475e5" containerName="payload-pre-processing"
Jul 29 04:35:58.957891 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:35:58.957844    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="57f84873-2bef-42fb-b337-e2da4bd475e5" containerName="payload-pre-processing"
Jul 29 04:35:58.958269 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:35:58.957966    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b6e291d-265a-4aa5-a74f-5763ee7294ae" containerName="manager"
Jul 29 04:35:58.958269 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:35:58.957977    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="4914f52b-c0da-4f3a-806d-d677e912c519" containerName="payload-processing"
Jul 29 04:35:58.958269 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:35:58.957986    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="063a7c97-fbf9-4f90-b1b9-30e38338716c" containerName="revoke-keys"
Jul 29 04:35:58.958269 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:35:58.957996    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="57f84873-2bef-42fb-b337-e2da4bd475e5" containerName="payload-pre-processing"
Jul 29 04:35:58.958269 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:35:58.958004    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="efbea725-8c24-4ec1-a591-ed159c827b2a" containerName="maas-api"
Jul 29 04:35:58.961012 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:35:58.960997    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wqt5k"
Jul 29 04:35:58.964435 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:35:58.964414    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 29 04:35:58.964990 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:35:58.964971    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-pc9k8\""
Jul 29 04:35:58.965112 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:35:58.965094    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 29 04:35:58.977842 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:35:58.977801    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wqt5k"]
Jul 29 04:35:59.121928 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:35:59.121890    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0879e712-d04d-496c-8fcd-6ef1fcce6bc1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-wqt5k\" (UID: \"0879e712-d04d-496c-8fcd-6ef1fcce6bc1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wqt5k"
Jul 29 04:35:59.122096 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:35:59.121939    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swnr6\" (UniqueName: \"kubernetes.io/projected/0879e712-d04d-496c-8fcd-6ef1fcce6bc1-kube-api-access-swnr6\") pod \"kuadrant-operator-controller-manager-86878c769f-wqt5k\" (UID: \"0879e712-d04d-496c-8fcd-6ef1fcce6bc1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wqt5k"
Jul 29 04:35:59.223264 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:35:59.223177    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swnr6\" (UniqueName: \"kubernetes.io/projected/0879e712-d04d-496c-8fcd-6ef1fcce6bc1-kube-api-access-swnr6\") pod \"kuadrant-operator-controller-manager-86878c769f-wqt5k\" (UID: \"0879e712-d04d-496c-8fcd-6ef1fcce6bc1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wqt5k"
Jul 29 04:35:59.223425 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:35:59.223319    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0879e712-d04d-496c-8fcd-6ef1fcce6bc1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-wqt5k\" (UID: \"0879e712-d04d-496c-8fcd-6ef1fcce6bc1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wqt5k"
Jul 29 04:35:59.223707 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:35:59.223686    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0879e712-d04d-496c-8fcd-6ef1fcce6bc1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-wqt5k\" (UID: \"0879e712-d04d-496c-8fcd-6ef1fcce6bc1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wqt5k"
Jul 29 04:35:59.232642 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:35:59.232611    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swnr6\" (UniqueName: \"kubernetes.io/projected/0879e712-d04d-496c-8fcd-6ef1fcce6bc1-kube-api-access-swnr6\") pod \"kuadrant-operator-controller-manager-86878c769f-wqt5k\" (UID: \"0879e712-d04d-496c-8fcd-6ef1fcce6bc1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wqt5k"
Jul 29 04:35:59.270638 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:35:59.270617    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wqt5k"
Jul 29 04:35:59.410166 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:35:59.410121    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wqt5k"]
Jul 29 04:35:59.412196 ip-10-0-135-138 kubenswrapper[2570]: W0729 04:35:59.412167    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0879e712_d04d_496c_8fcd_6ef1fcce6bc1.slice/crio-05f9ba481e90e2dbb435f9bfe9b9dd9d1a030e73a903884e02799dbf659c759e WatchSource:0}: Error finding container 05f9ba481e90e2dbb435f9bfe9b9dd9d1a030e73a903884e02799dbf659c759e: Status 404 returned error can't find the container with id 05f9ba481e90e2dbb435f9bfe9b9dd9d1a030e73a903884e02799dbf659c759e
Jul 29 04:35:59.662641 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:35:59.662597    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wqt5k" event={"ID":"0879e712-d04d-496c-8fcd-6ef1fcce6bc1","Type":"ContainerStarted","Data":"a50284c3d37e7470e4d32a0b82f272cd1068b09d438fee183df2933ed719e2f7"}
Jul 29 04:35:59.662807 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:35:59.662662    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wqt5k" event={"ID":"0879e712-d04d-496c-8fcd-6ef1fcce6bc1","Type":"ContainerStarted","Data":"05f9ba481e90e2dbb435f9bfe9b9dd9d1a030e73a903884e02799dbf659c759e"}
Jul 29 04:35:59.662879 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:35:59.662803    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wqt5k"
Jul 29 04:35:59.683375 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:35:59.683327    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wqt5k" podStartSLOduration=1.68331555 podStartE2EDuration="1.68331555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:35:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:35:59.6826467 +0000 UTC m=+2424.412777790" watchObservedRunningTime="2026-07-29 04:35:59.68331555 +0000 UTC m=+2424.413446619"
Jul 29 04:36:10.672891 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:36:10.672862    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wqt5k"
Jul 29 04:36:20.516235 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:36:20.516199    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-34b95582-666d79596b-drbls"]
Jul 29 04:36:20.520146 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:36:20.520126    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-34b95582-666d79596b-drbls"
Jul 29 04:36:20.523085 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:36:20.523053    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 29 04:36:20.524359 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:36:20.524332    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 29 04:36:20.524485 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:36:20.524330    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-34b95582-serving-cert\""
Jul 29 04:36:20.524485 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:36:20.524337    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-lh6p9\""
Jul 29 04:36:20.530773 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:36:20.530754    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-34b95582-666d79596b-drbls"]
Jul 29 04:36:20.613353 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:36:20.613321    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chqdl\" (UniqueName: \"kubernetes.io/projected/1d921f70-dce7-4893-ba0b-d4a85b84959d-kube-api-access-chqdl\") pod \"maas-api-e2e-mt-34b95582-666d79596b-drbls\" (UID: \"1d921f70-dce7-4893-ba0b-d4a85b84959d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-34b95582-666d79596b-drbls"
Jul 29 04:36:20.613509 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:36:20.613409    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d921f70-dce7-4893-ba0b-d4a85b84959d-openshift-service-ca\") pod \"maas-api-e2e-mt-34b95582-666d79596b-drbls\" (UID: \"1d921f70-dce7-4893-ba0b-d4a85b84959d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-34b95582-666d79596b-drbls"
Jul 29 04:36:20.613509 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:36:20.613440    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d921f70-dce7-4893-ba0b-d4a85b84959d-maas-api-tls\") pod \"maas-api-e2e-mt-34b95582-666d79596b-drbls\" (UID: \"1d921f70-dce7-4893-ba0b-d4a85b84959d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-34b95582-666d79596b-drbls"
Jul 29 04:36:20.714107 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:36:20.714075    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chqdl\" (UniqueName: \"kubernetes.io/projected/1d921f70-dce7-4893-ba0b-d4a85b84959d-kube-api-access-chqdl\") pod \"maas-api-e2e-mt-34b95582-666d79596b-drbls\" (UID: \"1d921f70-dce7-4893-ba0b-d4a85b84959d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-34b95582-666d79596b-drbls"
Jul 29 04:36:20.714267 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:36:20.714146    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d921f70-dce7-4893-ba0b-d4a85b84959d-openshift-service-ca\") pod \"maas-api-e2e-mt-34b95582-666d79596b-drbls\" (UID: \"1d921f70-dce7-4893-ba0b-d4a85b84959d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-34b95582-666d79596b-drbls"
Jul 29 04:36:20.714267 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:36:20.714177    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d921f70-dce7-4893-ba0b-d4a85b84959d-maas-api-tls\") pod \"maas-api-e2e-mt-34b95582-666d79596b-drbls\" (UID: \"1d921f70-dce7-4893-ba0b-d4a85b84959d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-34b95582-666d79596b-drbls"
Jul 29 04:36:20.714391 ip-10-0-135-138 kubenswrapper[2570]: E0729 04:36:20.714290    2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-34b95582-serving-cert: secret "maas-api-e2e-mt-34b95582-serving-cert" not found
Jul 29 04:36:20.714391 ip-10-0-135-138 kubenswrapper[2570]: E0729 04:36:20.714349    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d921f70-dce7-4893-ba0b-d4a85b84959d-maas-api-tls podName:1d921f70-dce7-4893-ba0b-d4a85b84959d nodeName:}" failed. No retries permitted until 2026-07-29 04:36:21.214328969 +0000 UTC m=+2445.944460031 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1d921f70-dce7-4893-ba0b-d4a85b84959d-maas-api-tls") pod "maas-api-e2e-mt-34b95582-666d79596b-drbls" (UID: "1d921f70-dce7-4893-ba0b-d4a85b84959d") : secret "maas-api-e2e-mt-34b95582-serving-cert" not found
Jul 29 04:36:20.714872 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:36:20.714850    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d921f70-dce7-4893-ba0b-d4a85b84959d-openshift-service-ca\") pod \"maas-api-e2e-mt-34b95582-666d79596b-drbls\" (UID: \"1d921f70-dce7-4893-ba0b-d4a85b84959d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-34b95582-666d79596b-drbls"
Jul 29 04:36:20.726642 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:36:20.726615    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chqdl\" (UniqueName: \"kubernetes.io/projected/1d921f70-dce7-4893-ba0b-d4a85b84959d-kube-api-access-chqdl\") pod \"maas-api-e2e-mt-34b95582-666d79596b-drbls\" (UID: \"1d921f70-dce7-4893-ba0b-d4a85b84959d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-34b95582-666d79596b-drbls"
Jul 29 04:36:21.218926 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:36:21.218895    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d921f70-dce7-4893-ba0b-d4a85b84959d-maas-api-tls\") pod \"maas-api-e2e-mt-34b95582-666d79596b-drbls\" (UID: \"1d921f70-dce7-4893-ba0b-d4a85b84959d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-34b95582-666d79596b-drbls"
Jul 29 04:36:21.221440 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:36:21.221419    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d921f70-dce7-4893-ba0b-d4a85b84959d-maas-api-tls\") pod \"maas-api-e2e-mt-34b95582-666d79596b-drbls\" (UID: \"1d921f70-dce7-4893-ba0b-d4a85b84959d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-34b95582-666d79596b-drbls"
Jul 29 04:36:21.321176 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:36:21.321145    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-34b95582-6fc6bb7dd4-lmq7g"]
Jul 29 04:36:21.325063 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:36:21.325041    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-34b95582-6fc6bb7dd4-lmq7g"
Jul 29 04:36:21.327858 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:36:21.327832    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-34b95582\""
Jul 29 04:36:21.327976 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:36:21.327871    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-34b95582-dockercfg-kxmv2\""
Jul 29 04:36:21.334316 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:36:21.334295    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-34b95582-6fc6bb7dd4-lmq7g"]
Jul 29 04:36:21.420329 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:36:21.420294    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mvtk\" (UniqueName: \"kubernetes.io/projected/e4ff21c5-9536-4fc6-ad8f-75beba1ac204-kube-api-access-8mvtk\") pod \"payload-pre-processing-e2e-mt-34b95582-6fc6bb7dd4-lmq7g\" (UID: \"e4ff21c5-9536-4fc6-ad8f-75beba1ac204\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-34b95582-6fc6bb7dd4-lmq7g"
Jul 29 04:36:21.420517 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:36:21.420350    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4ff21c5-9536-4fc6-ad8f-75beba1ac204-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-34b95582-6fc6bb7dd4-lmq7g\" (UID: \"e4ff21c5-9536-4fc6-ad8f-75beba1ac204\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-34b95582-6fc6bb7dd4-lmq7g"
Jul 29 04:36:21.422408 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:36:21.422383    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-34b95582-6b84fcccf4-l549t"]
Jul 29 04:36:21.425851 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:36:21.425814    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-34b95582-6b84fcccf4-l549t"
Jul 29 04:36:21.431059 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:36:21.431046    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-34b95582-666d79596b-drbls"
Jul 29 04:36:21.436488 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:36:21.436464    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-34b95582-6b84fcccf4-l549t"]
Jul 29 04:36:21.522291 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:36:21.521522    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mvtk\" (UniqueName: \"kubernetes.io/projected/e4ff21c5-9536-4fc6-ad8f-75beba1ac204-kube-api-access-8mvtk\") pod \"payload-pre-processing-e2e-mt-34b95582-6fc6bb7dd4-lmq7g\" (UID: \"e4ff21c5-9536-4fc6-ad8f-75beba1ac204\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-34b95582-6fc6bb7dd4-lmq7g"
Jul 29 04:36:21.522291 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:36:21.521572    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4ff21c5-9536-4fc6-ad8f-75beba1ac204-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-34b95582-6fc6bb7dd4-lmq7g\" (UID: \"e4ff21c5-9536-4fc6-ad8f-75beba1ac204\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-34b95582-6fc6bb7dd4-lmq7g"
Jul 29 04:36:21.522291 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:36:21.521649    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqrnj\" (UniqueName: \"kubernetes.io/projected/144ccc57-e84c-40ce-8588-7456ff7953f2-kube-api-access-sqrnj\") pod \"payload-processing-e2e-mt-34b95582-6b84fcccf4-l549t\" (UID: \"144ccc57-e84c-40ce-8588-7456ff7953f2\") " pod="openshift-ingress/payload-processing-e2e-mt-34b95582-6b84fcccf4-l549t"
Jul 29 04:36:21.522291 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:36:21.521683    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/144ccc57-e84c-40ce-8588-7456ff7953f2-plugins-config-volume\") pod \"payload-processing-e2e-mt-34b95582-6b84fcccf4-l549t\" (UID: \"144ccc57-e84c-40ce-8588-7456ff7953f2\") " pod="openshift-ingress/payload-processing-e2e-mt-34b95582-6b84fcccf4-l549t"
Jul 29 04:36:21.523163 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:36:21.522800    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4ff21c5-9536-4fc6-ad8f-75beba1ac204-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-34b95582-6fc6bb7dd4-lmq7g\" (UID: \"e4ff21c5-9536-4fc6-ad8f-75beba1ac204\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-34b95582-6fc6bb7dd4-lmq7g"
Jul 29 04:36:21.530752 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:36:21.530730    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mvtk\" (UniqueName: \"kubernetes.io/projected/e4ff21c5-9536-4fc6-ad8f-75beba1ac204-kube-api-access-8mvtk\") pod \"payload-pre-processing-e2e-mt-34b95582-6fc6bb7dd4-lmq7g\" (UID: \"e4ff21c5-9536-4fc6-ad8f-75beba1ac204\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-34b95582-6fc6bb7dd4-lmq7g"
Jul 29 04:36:21.575599 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:36:21.575573    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-34b95582-666d79596b-drbls"]
Jul 29 04:36:21.577485 ip-10-0-135-138 kubenswrapper[2570]: W0729 04:36:21.577459    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d921f70_dce7_4893_ba0b_d4a85b84959d.slice/crio-e2ebcbac4a1925d3b73c413a6420fde3da6bd8f8c4340d35c76bc23cf74a0b09 WatchSource:0}: Error finding container e2ebcbac4a1925d3b73c413a6420fde3da6bd8f8c4340d35c76bc23cf74a0b09: Status 404 returned error can't find the container with id e2ebcbac4a1925d3b73c413a6420fde3da6bd8f8c4340d35c76bc23cf74a0b09
Jul 29 04:36:21.622183 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:36:21.622153    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqrnj\" (UniqueName: \"kubernetes.io/projected/144ccc57-e84c-40ce-8588-7456ff7953f2-kube-api-access-sqrnj\") pod \"payload-processing-e2e-mt-34b95582-6b84fcccf4-l549t\" (UID: \"144ccc57-e84c-40ce-8588-7456ff7953f2\") " pod="openshift-ingress/payload-processing-e2e-mt-34b95582-6b84fcccf4-l549t"
Jul 29 04:36:21.622282 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:36:21.622186    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/144ccc57-e84c-40ce-8588-7456ff7953f2-plugins-config-volume\") pod \"payload-processing-e2e-mt-34b95582-6b84fcccf4-l549t\" (UID: \"144ccc57-e84c-40ce-8588-7456ff7953f2\") " pod="openshift-ingress/payload-processing-e2e-mt-34b95582-6b84fcccf4-l549t"
Jul 29 04:36:21.622868 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:36:21.622851    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/144ccc57-e84c-40ce-8588-7456ff7953f2-plugins-config-volume\") pod \"payload-processing-e2e-mt-34b95582-6b84fcccf4-l549t\" (UID: \"144ccc57-e84c-40ce-8588-7456ff7953f2\") " pod="openshift-ingress/payload-processing-e2e-mt-34b95582-6b84fcccf4-l549t"
Jul 29 04:36:21.631452 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:36:21.631424    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqrnj\" (UniqueName: \"kubernetes.io/projected/144ccc57-e84c-40ce-8588-7456ff7953f2-kube-api-access-sqrnj\") pod \"payload-processing-e2e-mt-34b95582-6b84fcccf4-l549t\" (UID: \"144ccc57-e84c-40ce-8588-7456ff7953f2\") " pod="openshift-ingress/payload-processing-e2e-mt-34b95582-6b84fcccf4-l549t"
Jul 29 04:36:21.637244 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:36:21.637228    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-34b95582-6fc6bb7dd4-lmq7g"
Jul 29 04:36:21.737418 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:36:21.737342    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-34b95582-6b84fcccf4-l549t"
Jul 29 04:36:21.772952 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:36:21.772932    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-34b95582-6fc6bb7dd4-lmq7g"]
Jul 29 04:36:21.779206 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:36:21.779070    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-34b95582-666d79596b-drbls" event={"ID":"1d921f70-dce7-4893-ba0b-d4a85b84959d","Type":"ContainerStarted","Data":"e2ebcbac4a1925d3b73c413a6420fde3da6bd8f8c4340d35c76bc23cf74a0b09"}
Jul 29 04:36:21.779478 ip-10-0-135-138 kubenswrapper[2570]: W0729 04:36:21.779446    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4ff21c5_9536_4fc6_ad8f_75beba1ac204.slice/crio-770ce2aee4de1600b9c50085df4b3f6ef3b6ca9176f95c721f1e5b1e97ccb1c8 WatchSource:0}: Error finding container 770ce2aee4de1600b9c50085df4b3f6ef3b6ca9176f95c721f1e5b1e97ccb1c8: Status 404 returned error can't find the container with id 770ce2aee4de1600b9c50085df4b3f6ef3b6ca9176f95c721f1e5b1e97ccb1c8
Jul 29 04:36:21.885535 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:36:21.885509    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-34b95582-6b84fcccf4-l549t"]
Jul 29 04:36:21.887209 ip-10-0-135-138 kubenswrapper[2570]: W0729 04:36:21.887182    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod144ccc57_e84c_40ce_8588_7456ff7953f2.slice/crio-4d468ef5388fa65d0b34ce4e7ad35b3bf5751730a71f9124a4f0e3aac5908696 WatchSource:0}: Error finding container 4d468ef5388fa65d0b34ce4e7ad35b3bf5751730a71f9124a4f0e3aac5908696: Status 404 returned error can't find the container with id 4d468ef5388fa65d0b34ce4e7ad35b3bf5751730a71f9124a4f0e3aac5908696
Jul 29 04:36:22.786365 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:36:22.786334    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-34b95582-666d79596b-drbls" event={"ID":"1d921f70-dce7-4893-ba0b-d4a85b84959d","Type":"ContainerStarted","Data":"cd69c6be2bdcc1a0403f7f24d8786ccaaaea3910920416ce4dd04fe928765af3"}
Jul 29 04:36:22.786811 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:36:22.786397    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-34b95582-666d79596b-drbls"
Jul 29 04:36:22.787899 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:36:22.787876    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-34b95582-6b84fcccf4-l549t" event={"ID":"144ccc57-e84c-40ce-8588-7456ff7953f2","Type":"ContainerStarted","Data":"52f25031d78f93a1f01c3590eff246755a3be4b21e04ce11b9375d949f975f86"}
Jul 29 04:36:22.787987 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:36:22.787925    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-34b95582-6b84fcccf4-l549t" event={"ID":"144ccc57-e84c-40ce-8588-7456ff7953f2","Type":"ContainerStarted","Data":"4d468ef5388fa65d0b34ce4e7ad35b3bf5751730a71f9124a4f0e3aac5908696"}
Jul 29 04:36:22.787987 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:36:22.787946    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-34b95582-6b84fcccf4-l549t"
Jul 29 04:36:22.789258 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:36:22.789239    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-34b95582-6fc6bb7dd4-lmq7g" event={"ID":"e4ff21c5-9536-4fc6-ad8f-75beba1ac204","Type":"ContainerStarted","Data":"0d8aebdc8c26dcbbe777f4c0edddaac4c7fe8d80e5683ab3ce7d5f7de3d5c240"}
Jul 29 04:36:22.789318 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:36:22.789279    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-34b95582-6fc6bb7dd4-lmq7g"
Jul 29 04:36:22.789318 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:36:22.789297    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-34b95582-6fc6bb7dd4-lmq7g" event={"ID":"e4ff21c5-9536-4fc6-ad8f-75beba1ac204","Type":"ContainerStarted","Data":"770ce2aee4de1600b9c50085df4b3f6ef3b6ca9176f95c721f1e5b1e97ccb1c8"}
Jul 29 04:36:22.807576 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:36:22.807515    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-34b95582-666d79596b-drbls" podStartSLOduration=2.390111747 podStartE2EDuration="2.807497547s" totalImagesPullingTime="417.3858ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:36:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:36:22.805970919 +0000 UTC m=+2447.536102012" watchObservedRunningTime="2026-07-29 04:36:22.807497547 +0000 UTC m=+2447.537628619"
Jul 29 04:36:22.826639 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:36:22.826598    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-34b95582-6fc6bb7dd4-lmq7g" podStartSLOduration=1.826584527 podStartE2EDuration="1.826584527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:36:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:36:22.824663375 +0000 UTC m=+2447.554794474" watchObservedRunningTime="2026-07-29 04:36:22.826584527 +0000 UTC m=+2447.556715597"
Jul 29 04:36:22.843926 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:36:22.843884    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-34b95582-6b84fcccf4-l549t" podStartSLOduration=1.843868872 podStartE2EDuration="1.843868872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:36:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:36:22.842127964 +0000 UTC m=+2447.572259052" watchObservedRunningTime="2026-07-29 04:36:22.843868872 +0000 UTC m=+2447.573999941"
Jul 29 04:36:28.799297 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:36:28.799266    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-34b95582-666d79596b-drbls"
Jul 29 04:36:33.795471 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:36:33.795432    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-34b95582-6b84fcccf4-l549t"
Jul 29 04:36:33.796048 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:36:33.795685    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-34b95582-6fc6bb7dd4-lmq7g"
Jul 29 04:37:19.601229 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:19.601193    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-34b95582-r9rss"]
Jul 29 04:37:19.604792 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:19.604775    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-34b95582-r9rss"
Jul 29 04:37:19.607658 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:19.607636    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8frm\""
Jul 29 04:37:19.612668 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:19.612492    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-34b95582-r9rss"]
Jul 29 04:37:19.662133 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:19.662100    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1c84cea-8a82-4692-8a0a-7bef68aaa4f3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-34b95582-r9rss\" (UID: \"d1c84cea-8a82-4692-8a0a-7bef68aaa4f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-34b95582-r9rss"
Jul 29 04:37:19.763011 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:19.762974    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1c84cea-8a82-4692-8a0a-7bef68aaa4f3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-34b95582-r9rss\" (UID: \"d1c84cea-8a82-4692-8a0a-7bef68aaa4f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-34b95582-r9rss"
Jul 29 04:37:19.763620 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:19.763601    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1c84cea-8a82-4692-8a0a-7bef68aaa4f3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-34b95582-r9rss\" (UID: \"d1c84cea-8a82-4692-8a0a-7bef68aaa4f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-34b95582-r9rss"
Jul 29 04:37:19.916262 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:19.916178    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-34b95582-r9rss"
Jul 29 04:37:20.057239 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:20.057212    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-34b95582-r9rss"]
Jul 29 04:37:20.059445 ip-10-0-135-138 kubenswrapper[2570]: W0729 04:37:20.059410    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1c84cea_8a82_4692_8a0a_7bef68aaa4f3.slice/crio-e9d82ce1b48a71ccce1ed91258c991b8b74829e71df9a3418d1844e9990505e3 WatchSource:0}: Error finding container e9d82ce1b48a71ccce1ed91258c991b8b74829e71df9a3418d1844e9990505e3: Status 404 returned error can't find the container with id e9d82ce1b48a71ccce1ed91258c991b8b74829e71df9a3418d1844e9990505e3
Jul 29 04:37:20.072522 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:20.072498    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-34b95582-r9rss" event={"ID":"d1c84cea-8a82-4692-8a0a-7bef68aaa4f3","Type":"ContainerStarted","Data":"e9d82ce1b48a71ccce1ed91258c991b8b74829e71df9a3418d1844e9990505e3"}
Jul 29 04:37:21.078344 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:21.078255    2570 generic.go:356] "Generic (PLEG): container finished" podID="d1c84cea-8a82-4692-8a0a-7bef68aaa4f3" containerID="e80fddcd2acc4e4657d953aca7b28a93c2024c7a1112d84739a63c5574c66463" exitCode=0
Jul 29 04:37:21.078344 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:21.078316    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-34b95582-r9rss" event={"ID":"d1c84cea-8a82-4692-8a0a-7bef68aaa4f3","Type":"ContainerDied","Data":"e80fddcd2acc4e4657d953aca7b28a93c2024c7a1112d84739a63c5574c66463"}
Jul 29 04:37:22.212903 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:22.212882    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-34b95582-r9rss"
Jul 29 04:37:22.291744 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:22.291715    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1c84cea-8a82-4692-8a0a-7bef68aaa4f3-maas-api-service-ca\") pod \"d1c84cea-8a82-4692-8a0a-7bef68aaa4f3\" (UID: \"d1c84cea-8a82-4692-8a0a-7bef68aaa4f3\") "
Jul 29 04:37:22.292060 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:22.292039    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1c84cea-8a82-4692-8a0a-7bef68aaa4f3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d1c84cea-8a82-4692-8a0a-7bef68aaa4f3" (UID: "d1c84cea-8a82-4692-8a0a-7bef68aaa4f3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 04:37:22.393360 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:22.393293    2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1c84cea-8a82-4692-8a0a-7bef68aaa4f3-maas-api-service-ca\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:37:23.089269 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:23.089237    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-34b95582-r9rss"
Jul 29 04:37:23.089448 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:23.089284    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-34b95582-r9rss" event={"ID":"d1c84cea-8a82-4692-8a0a-7bef68aaa4f3","Type":"ContainerDied","Data":"e9d82ce1b48a71ccce1ed91258c991b8b74829e71df9a3418d1844e9990505e3"}
Jul 29 04:37:23.089448 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:23.089331    2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9d82ce1b48a71ccce1ed91258c991b8b74829e71df9a3418d1844e9990505e3"
Jul 29 04:37:35.134082 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:35.134000    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-34b95582-666d79596b-drbls"]
Jul 29 04:37:35.134517 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:35.134206    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-34b95582-666d79596b-drbls" podUID="1d921f70-dce7-4893-ba0b-d4a85b84959d" containerName="maas-api" containerID="cri-o://cd69c6be2bdcc1a0403f7f24d8786ccaaaea3910920416ce4dd04fe928765af3" gracePeriod=30
Jul 29 04:37:35.186482 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:35.186454    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-34b95582-6b84fcccf4-l549t"]
Jul 29 04:37:35.186715 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:35.186655    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-34b95582-6b84fcccf4-l549t" podUID="144ccc57-e84c-40ce-8588-7456ff7953f2" containerName="payload-processing" containerID="cri-o://52f25031d78f93a1f01c3590eff246755a3be4b21e04ce11b9375d949f975f86" gracePeriod=30
Jul 29 04:37:35.194594 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:35.194571    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-34b95582-6fc6bb7dd4-lmq7g"]
Jul 29 04:37:35.194754 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:35.194737    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-34b95582-6fc6bb7dd4-lmq7g" podUID="e4ff21c5-9536-4fc6-ad8f-75beba1ac204" containerName="payload-pre-processing" containerID="cri-o://0d8aebdc8c26dcbbe777f4c0edddaac4c7fe8d80e5683ab3ce7d5f7de3d5c240" gracePeriod=30
Jul 29 04:37:35.339493 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:35.339446    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-34b95582-r9rss"]
Jul 29 04:37:35.341335 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:35.341307    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-34b95582-r9rss"]
Jul 29 04:37:35.447138 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:35.447116    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-34b95582-666d79596b-drbls"
Jul 29 04:37:35.489012 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:35.488990    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-34b95582-6b84fcccf4-l549t"
Jul 29 04:37:35.491920 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:35.491898    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-34b95582-6fc6bb7dd4-lmq7g"
Jul 29 04:37:35.513623 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:35.513594    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-chqdl\" (UniqueName: \"kubernetes.io/projected/1d921f70-dce7-4893-ba0b-d4a85b84959d-kube-api-access-chqdl\") pod \"1d921f70-dce7-4893-ba0b-d4a85b84959d\" (UID: \"1d921f70-dce7-4893-ba0b-d4a85b84959d\") "
Jul 29 04:37:35.513757 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:35.513651    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4ff21c5-9536-4fc6-ad8f-75beba1ac204-plugins-config-volume\") pod \"e4ff21c5-9536-4fc6-ad8f-75beba1ac204\" (UID: \"e4ff21c5-9536-4fc6-ad8f-75beba1ac204\") "
Jul 29 04:37:35.513757 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:35.513681    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8mvtk\" (UniqueName: \"kubernetes.io/projected/e4ff21c5-9536-4fc6-ad8f-75beba1ac204-kube-api-access-8mvtk\") pod \"e4ff21c5-9536-4fc6-ad8f-75beba1ac204\" (UID: \"e4ff21c5-9536-4fc6-ad8f-75beba1ac204\") "
Jul 29 04:37:35.513757 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:35.513710    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sqrnj\" (UniqueName: \"kubernetes.io/projected/144ccc57-e84c-40ce-8588-7456ff7953f2-kube-api-access-sqrnj\") pod \"144ccc57-e84c-40ce-8588-7456ff7953f2\" (UID: \"144ccc57-e84c-40ce-8588-7456ff7953f2\") "
Jul 29 04:37:35.513951 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:35.513755    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/144ccc57-e84c-40ce-8588-7456ff7953f2-plugins-config-volume\") pod \"144ccc57-e84c-40ce-8588-7456ff7953f2\" (UID: \"144ccc57-e84c-40ce-8588-7456ff7953f2\") "
Jul 29 04:37:35.513951 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:35.513886    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d921f70-dce7-4893-ba0b-d4a85b84959d-openshift-service-ca\") pod \"1d921f70-dce7-4893-ba0b-d4a85b84959d\" (UID: \"1d921f70-dce7-4893-ba0b-d4a85b84959d\") "
Jul 29 04:37:35.513951 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:35.513916    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d921f70-dce7-4893-ba0b-d4a85b84959d-maas-api-tls\") pod \"1d921f70-dce7-4893-ba0b-d4a85b84959d\" (UID: \"1d921f70-dce7-4893-ba0b-d4a85b84959d\") "
Jul 29 04:37:35.514795 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:35.514581    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d921f70-dce7-4893-ba0b-d4a85b84959d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1d921f70-dce7-4893-ba0b-d4a85b84959d" (UID: "1d921f70-dce7-4893-ba0b-d4a85b84959d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 04:37:35.514795 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:35.514650    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4ff21c5-9536-4fc6-ad8f-75beba1ac204-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e4ff21c5-9536-4fc6-ad8f-75beba1ac204" (UID: "e4ff21c5-9536-4fc6-ad8f-75beba1ac204"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 04:37:35.514795 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:35.514767    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/144ccc57-e84c-40ce-8588-7456ff7953f2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "144ccc57-e84c-40ce-8588-7456ff7953f2" (UID: "144ccc57-e84c-40ce-8588-7456ff7953f2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 04:37:35.516630 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:35.516600    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d921f70-dce7-4893-ba0b-d4a85b84959d-kube-api-access-chqdl" (OuterVolumeSpecName: "kube-api-access-chqdl") pod "1d921f70-dce7-4893-ba0b-d4a85b84959d" (UID: "1d921f70-dce7-4893-ba0b-d4a85b84959d"). InnerVolumeSpecName "kube-api-access-chqdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 04:37:35.516746 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:35.516689    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4ff21c5-9536-4fc6-ad8f-75beba1ac204-kube-api-access-8mvtk" (OuterVolumeSpecName: "kube-api-access-8mvtk") pod "e4ff21c5-9536-4fc6-ad8f-75beba1ac204" (UID: "e4ff21c5-9536-4fc6-ad8f-75beba1ac204"). InnerVolumeSpecName "kube-api-access-8mvtk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 04:37:35.517224 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:35.517198    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/144ccc57-e84c-40ce-8588-7456ff7953f2-kube-api-access-sqrnj" (OuterVolumeSpecName: "kube-api-access-sqrnj") pod "144ccc57-e84c-40ce-8588-7456ff7953f2" (UID: "144ccc57-e84c-40ce-8588-7456ff7953f2"). InnerVolumeSpecName "kube-api-access-sqrnj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 04:37:35.517324 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:35.517270    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d921f70-dce7-4893-ba0b-d4a85b84959d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1d921f70-dce7-4893-ba0b-d4a85b84959d" (UID: "1d921f70-dce7-4893-ba0b-d4a85b84959d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 04:37:35.614724 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:35.614693    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chqdl\" (UniqueName: \"kubernetes.io/projected/1d921f70-dce7-4893-ba0b-d4a85b84959d-kube-api-access-chqdl\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:37:35.614724 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:35.614721    2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4ff21c5-9536-4fc6-ad8f-75beba1ac204-plugins-config-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:37:35.614948 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:35.614731    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mvtk\" (UniqueName: \"kubernetes.io/projected/e4ff21c5-9536-4fc6-ad8f-75beba1ac204-kube-api-access-8mvtk\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:37:35.614948 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:35.614741    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqrnj\" (UniqueName: \"kubernetes.io/projected/144ccc57-e84c-40ce-8588-7456ff7953f2-kube-api-access-sqrnj\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:37:35.614948 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:35.614750    2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/144ccc57-e84c-40ce-8588-7456ff7953f2-plugins-config-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:37:35.614948 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:35.614758    2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d921f70-dce7-4893-ba0b-d4a85b84959d-openshift-service-ca\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:37:35.614948 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:35.614768    2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d921f70-dce7-4893-ba0b-d4a85b84959d-maas-api-tls\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:37:35.836017 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:35.835986    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1c84cea-8a82-4692-8a0a-7bef68aaa4f3" path="/var/lib/kubelet/pods/d1c84cea-8a82-4692-8a0a-7bef68aaa4f3/volumes"
Jul 29 04:37:36.157553 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:36.157520    2570 generic.go:356] "Generic (PLEG): container finished" podID="1d921f70-dce7-4893-ba0b-d4a85b84959d" containerID="cd69c6be2bdcc1a0403f7f24d8786ccaaaea3910920416ce4dd04fe928765af3" exitCode=0
Jul 29 04:37:36.157981 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:36.157567    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-34b95582-666d79596b-drbls"
Jul 29 04:37:36.157981 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:36.157595    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-34b95582-666d79596b-drbls" event={"ID":"1d921f70-dce7-4893-ba0b-d4a85b84959d","Type":"ContainerDied","Data":"cd69c6be2bdcc1a0403f7f24d8786ccaaaea3910920416ce4dd04fe928765af3"}
Jul 29 04:37:36.157981 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:36.157659    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-34b95582-666d79596b-drbls" event={"ID":"1d921f70-dce7-4893-ba0b-d4a85b84959d","Type":"ContainerDied","Data":"e2ebcbac4a1925d3b73c413a6420fde3da6bd8f8c4340d35c76bc23cf74a0b09"}
Jul 29 04:37:36.157981 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:36.157690    2570 scope.go:117] "RemoveContainer" containerID="cd69c6be2bdcc1a0403f7f24d8786ccaaaea3910920416ce4dd04fe928765af3"
Jul 29 04:37:36.159124 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:36.159100    2570 generic.go:356] "Generic (PLEG): container finished" podID="144ccc57-e84c-40ce-8588-7456ff7953f2" containerID="52f25031d78f93a1f01c3590eff246755a3be4b21e04ce11b9375d949f975f86" exitCode=0
Jul 29 04:37:36.159227 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:36.159151    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-34b95582-6b84fcccf4-l549t"
Jul 29 04:37:36.159227 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:36.159189    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-34b95582-6b84fcccf4-l549t" event={"ID":"144ccc57-e84c-40ce-8588-7456ff7953f2","Type":"ContainerDied","Data":"52f25031d78f93a1f01c3590eff246755a3be4b21e04ce11b9375d949f975f86"}
Jul 29 04:37:36.159352 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:36.159240    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-34b95582-6b84fcccf4-l549t" event={"ID":"144ccc57-e84c-40ce-8588-7456ff7953f2","Type":"ContainerDied","Data":"4d468ef5388fa65d0b34ce4e7ad35b3bf5751730a71f9124a4f0e3aac5908696"}
Jul 29 04:37:36.160662 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:36.160640    2570 generic.go:356] "Generic (PLEG): container finished" podID="e4ff21c5-9536-4fc6-ad8f-75beba1ac204" containerID="0d8aebdc8c26dcbbe777f4c0edddaac4c7fe8d80e5683ab3ce7d5f7de3d5c240" exitCode=0
Jul 29 04:37:36.160776 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:36.160670    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-34b95582-6fc6bb7dd4-lmq7g" event={"ID":"e4ff21c5-9536-4fc6-ad8f-75beba1ac204","Type":"ContainerDied","Data":"0d8aebdc8c26dcbbe777f4c0edddaac4c7fe8d80e5683ab3ce7d5f7de3d5c240"}
Jul 29 04:37:36.160776 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:36.160701    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-34b95582-6fc6bb7dd4-lmq7g"
Jul 29 04:37:36.160776 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:36.160726    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-34b95582-6fc6bb7dd4-lmq7g" event={"ID":"e4ff21c5-9536-4fc6-ad8f-75beba1ac204","Type":"ContainerDied","Data":"770ce2aee4de1600b9c50085df4b3f6ef3b6ca9176f95c721f1e5b1e97ccb1c8"}
Jul 29 04:37:36.167215 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:36.167191    2570 scope.go:117] "RemoveContainer" containerID="cd69c6be2bdcc1a0403f7f24d8786ccaaaea3910920416ce4dd04fe928765af3"
Jul 29 04:37:36.167464 ip-10-0-135-138 kubenswrapper[2570]: E0729 04:37:36.167443    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd69c6be2bdcc1a0403f7f24d8786ccaaaea3910920416ce4dd04fe928765af3\": container with ID starting with cd69c6be2bdcc1a0403f7f24d8786ccaaaea3910920416ce4dd04fe928765af3 not found: ID does not exist" containerID="cd69c6be2bdcc1a0403f7f24d8786ccaaaea3910920416ce4dd04fe928765af3"
Jul 29 04:37:36.167543 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:36.167485    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd69c6be2bdcc1a0403f7f24d8786ccaaaea3910920416ce4dd04fe928765af3"} err="failed to get container status \"cd69c6be2bdcc1a0403f7f24d8786ccaaaea3910920416ce4dd04fe928765af3\": rpc error: code = NotFound desc = could not find container \"cd69c6be2bdcc1a0403f7f24d8786ccaaaea3910920416ce4dd04fe928765af3\": container with ID starting with cd69c6be2bdcc1a0403f7f24d8786ccaaaea3910920416ce4dd04fe928765af3 not found: ID does not exist"
Jul 29 04:37:36.167543 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:36.167504    2570 scope.go:117] "RemoveContainer" containerID="52f25031d78f93a1f01c3590eff246755a3be4b21e04ce11b9375d949f975f86"
Jul 29 04:37:36.174967 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:36.174953    2570 scope.go:117] "RemoveContainer" containerID="52f25031d78f93a1f01c3590eff246755a3be4b21e04ce11b9375d949f975f86"
Jul 29 04:37:36.175195 ip-10-0-135-138 kubenswrapper[2570]: E0729 04:37:36.175177    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52f25031d78f93a1f01c3590eff246755a3be4b21e04ce11b9375d949f975f86\": container with ID starting with 52f25031d78f93a1f01c3590eff246755a3be4b21e04ce11b9375d949f975f86 not found: ID does not exist" containerID="52f25031d78f93a1f01c3590eff246755a3be4b21e04ce11b9375d949f975f86"
Jul 29 04:37:36.175250 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:36.175205    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52f25031d78f93a1f01c3590eff246755a3be4b21e04ce11b9375d949f975f86"} err="failed to get container status \"52f25031d78f93a1f01c3590eff246755a3be4b21e04ce11b9375d949f975f86\": rpc error: code = NotFound desc = could not find container \"52f25031d78f93a1f01c3590eff246755a3be4b21e04ce11b9375d949f975f86\": container with ID starting with 52f25031d78f93a1f01c3590eff246755a3be4b21e04ce11b9375d949f975f86 not found: ID does not exist"
Jul 29 04:37:36.175250 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:36.175227    2570 scope.go:117] "RemoveContainer" containerID="0d8aebdc8c26dcbbe777f4c0edddaac4c7fe8d80e5683ab3ce7d5f7de3d5c240"
Jul 29 04:37:36.181355 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:36.181334    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-34b95582-6b84fcccf4-l549t"]
Jul 29 04:37:36.183329 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:36.183309    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-34b95582-6b84fcccf4-l549t"]
Jul 29 04:37:36.184617 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:36.184587    2570 scope.go:117] "RemoveContainer" containerID="0d8aebdc8c26dcbbe777f4c0edddaac4c7fe8d80e5683ab3ce7d5f7de3d5c240"
Jul 29 04:37:36.184945 ip-10-0-135-138 kubenswrapper[2570]: E0729 04:37:36.184897    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d8aebdc8c26dcbbe777f4c0edddaac4c7fe8d80e5683ab3ce7d5f7de3d5c240\": container with ID starting with 0d8aebdc8c26dcbbe777f4c0edddaac4c7fe8d80e5683ab3ce7d5f7de3d5c240 not found: ID does not exist" containerID="0d8aebdc8c26dcbbe777f4c0edddaac4c7fe8d80e5683ab3ce7d5f7de3d5c240"
Jul 29 04:37:36.185109 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:36.185045    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d8aebdc8c26dcbbe777f4c0edddaac4c7fe8d80e5683ab3ce7d5f7de3d5c240"} err="failed to get container status \"0d8aebdc8c26dcbbe777f4c0edddaac4c7fe8d80e5683ab3ce7d5f7de3d5c240\": rpc error: code = NotFound desc = could not find container \"0d8aebdc8c26dcbbe777f4c0edddaac4c7fe8d80e5683ab3ce7d5f7de3d5c240\": container with ID starting with 0d8aebdc8c26dcbbe777f4c0edddaac4c7fe8d80e5683ab3ce7d5f7de3d5c240 not found: ID does not exist"
Jul 29 04:37:36.192318 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:36.192297    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-34b95582-666d79596b-drbls"]
Jul 29 04:37:36.195579 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:36.195562    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-34b95582-666d79596b-drbls"]
Jul 29 04:37:36.205954 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:36.205889    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-34b95582-6fc6bb7dd4-lmq7g"]
Jul 29 04:37:36.208108 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:36.208080    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-34b95582-6fc6bb7dd4-lmq7g"]
Jul 29 04:37:37.835294 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:37.835262    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="144ccc57-e84c-40ce-8588-7456ff7953f2" path="/var/lib/kubelet/pods/144ccc57-e84c-40ce-8588-7456ff7953f2/volumes"
Jul 29 04:37:37.835655 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:37.835643    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d921f70-dce7-4893-ba0b-d4a85b84959d" path="/var/lib/kubelet/pods/1d921f70-dce7-4893-ba0b-d4a85b84959d/volumes"
Jul 29 04:37:37.835979 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:37:37.835965    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4ff21c5-9536-4fc6-ad8f-75beba1ac204" path="/var/lib/kubelet/pods/e4ff21c5-9536-4fc6-ad8f-75beba1ac204/volumes"
Jul 29 04:40:29.647799 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:29.647718    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1fcfbd-78c65659b8-jtr2j"]
Jul 29 04:40:29.648340 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:29.648119    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="144ccc57-e84c-40ce-8588-7456ff7953f2" containerName="payload-processing"
Jul 29 04:40:29.648340 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:29.648130    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="144ccc57-e84c-40ce-8588-7456ff7953f2" containerName="payload-processing"
Jul 29 04:40:29.648340 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:29.648141    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d921f70-dce7-4893-ba0b-d4a85b84959d" containerName="maas-api"
Jul 29 04:40:29.648340 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:29.648146    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d921f70-dce7-4893-ba0b-d4a85b84959d" containerName="maas-api"
Jul 29 04:40:29.648340 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:29.648158    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4ff21c5-9536-4fc6-ad8f-75beba1ac204" containerName="payload-pre-processing"
Jul 29 04:40:29.648340 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:29.648163    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4ff21c5-9536-4fc6-ad8f-75beba1ac204" containerName="payload-pre-processing"
Jul 29 04:40:29.648340 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:29.648169    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1c84cea-8a82-4692-8a0a-7bef68aaa4f3" containerName="revoke-keys"
Jul 29 04:40:29.648340 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:29.648174    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1c84cea-8a82-4692-8a0a-7bef68aaa4f3" containerName="revoke-keys"
Jul 29 04:40:29.648340 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:29.648249    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1c84cea-8a82-4692-8a0a-7bef68aaa4f3" containerName="revoke-keys"
Jul 29 04:40:29.648340 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:29.648261    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4ff21c5-9536-4fc6-ad8f-75beba1ac204" containerName="payload-pre-processing"
Jul 29 04:40:29.648340 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:29.648271    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="144ccc57-e84c-40ce-8588-7456ff7953f2" containerName="payload-processing"
Jul 29 04:40:29.648340 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:29.648284    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d921f70-dce7-4893-ba0b-d4a85b84959d" containerName="maas-api"
Jul 29 04:40:29.651262 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:29.651245    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fcfbd-78c65659b8-jtr2j"
Jul 29 04:40:29.654393 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:29.654370    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-1fcfbd-serving-cert\""
Jul 29 04:40:29.654498 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:29.654377    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 29 04:40:29.654616 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:29.654603    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 29 04:40:29.655580 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:29.655565    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-lh6p9\""
Jul 29 04:40:29.668220 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:29.668178    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1fcfbd-78c65659b8-jtr2j"]
Jul 29 04:40:29.784118 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:29.784085    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhjjl\" (UniqueName: \"kubernetes.io/projected/7e6ccb39-07ea-45e1-ad39-5da90124c5ec-kube-api-access-xhjjl\") pod \"maas-api-e2e-inf-a-1fcfbd-78c65659b8-jtr2j\" (UID: \"7e6ccb39-07ea-45e1-ad39-5da90124c5ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fcfbd-78c65659b8-jtr2j"
Jul 29 04:40:29.784279 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:29.784124    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e6ccb39-07ea-45e1-ad39-5da90124c5ec-maas-api-tls\") pod \"maas-api-e2e-inf-a-1fcfbd-78c65659b8-jtr2j\" (UID: \"7e6ccb39-07ea-45e1-ad39-5da90124c5ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fcfbd-78c65659b8-jtr2j"
Jul 29 04:40:29.784279 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:29.784143    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e6ccb39-07ea-45e1-ad39-5da90124c5ec-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1fcfbd-78c65659b8-jtr2j\" (UID: \"7e6ccb39-07ea-45e1-ad39-5da90124c5ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fcfbd-78c65659b8-jtr2j"
Jul 29 04:40:29.884717 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:29.884686    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhjjl\" (UniqueName: \"kubernetes.io/projected/7e6ccb39-07ea-45e1-ad39-5da90124c5ec-kube-api-access-xhjjl\") pod \"maas-api-e2e-inf-a-1fcfbd-78c65659b8-jtr2j\" (UID: \"7e6ccb39-07ea-45e1-ad39-5da90124c5ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fcfbd-78c65659b8-jtr2j"
Jul 29 04:40:29.884915 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:29.884726    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e6ccb39-07ea-45e1-ad39-5da90124c5ec-maas-api-tls\") pod \"maas-api-e2e-inf-a-1fcfbd-78c65659b8-jtr2j\" (UID: \"7e6ccb39-07ea-45e1-ad39-5da90124c5ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fcfbd-78c65659b8-jtr2j"
Jul 29 04:40:29.884915 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:29.884744    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e6ccb39-07ea-45e1-ad39-5da90124c5ec-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1fcfbd-78c65659b8-jtr2j\" (UID: \"7e6ccb39-07ea-45e1-ad39-5da90124c5ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fcfbd-78c65659b8-jtr2j"
Jul 29 04:40:29.884915 ip-10-0-135-138 kubenswrapper[2570]: E0729 04:40:29.884870    2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-1fcfbd-serving-cert: secret "maas-api-e2e-inf-a-1fcfbd-serving-cert" not found
Jul 29 04:40:29.885040 ip-10-0-135-138 kubenswrapper[2570]: E0729 04:40:29.884951    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e6ccb39-07ea-45e1-ad39-5da90124c5ec-maas-api-tls podName:7e6ccb39-07ea-45e1-ad39-5da90124c5ec nodeName:}" failed. No retries permitted until 2026-07-29 04:40:30.384930637 +0000 UTC m=+2695.115061689 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7e6ccb39-07ea-45e1-ad39-5da90124c5ec-maas-api-tls") pod "maas-api-e2e-inf-a-1fcfbd-78c65659b8-jtr2j" (UID: "7e6ccb39-07ea-45e1-ad39-5da90124c5ec") : secret "maas-api-e2e-inf-a-1fcfbd-serving-cert" not found
Jul 29 04:40:29.885322 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:29.885305    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e6ccb39-07ea-45e1-ad39-5da90124c5ec-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1fcfbd-78c65659b8-jtr2j\" (UID: \"7e6ccb39-07ea-45e1-ad39-5da90124c5ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fcfbd-78c65659b8-jtr2j"
Jul 29 04:40:29.895047 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:29.895026    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhjjl\" (UniqueName: \"kubernetes.io/projected/7e6ccb39-07ea-45e1-ad39-5da90124c5ec-kube-api-access-xhjjl\") pod \"maas-api-e2e-inf-a-1fcfbd-78c65659b8-jtr2j\" (UID: \"7e6ccb39-07ea-45e1-ad39-5da90124c5ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fcfbd-78c65659b8-jtr2j"
Jul 29 04:40:30.390268 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:30.390234    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e6ccb39-07ea-45e1-ad39-5da90124c5ec-maas-api-tls\") pod \"maas-api-e2e-inf-a-1fcfbd-78c65659b8-jtr2j\" (UID: \"7e6ccb39-07ea-45e1-ad39-5da90124c5ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fcfbd-78c65659b8-jtr2j"
Jul 29 04:40:30.392792 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:30.392766    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e6ccb39-07ea-45e1-ad39-5da90124c5ec-maas-api-tls\") pod \"maas-api-e2e-inf-a-1fcfbd-78c65659b8-jtr2j\" (UID: \"7e6ccb39-07ea-45e1-ad39-5da90124c5ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fcfbd-78c65659b8-jtr2j"
Jul 29 04:40:30.403582 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:30.403559    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1fcfbd-c6985bd7b-d49sr"]
Jul 29 04:40:30.407264 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:30.407247    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1fcfbd-c6985bd7b-d49sr"
Jul 29 04:40:30.410067 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:30.410046    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-1fcfbd\""
Jul 29 04:40:30.418754 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:30.418733    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1fcfbd-c6985bd7b-d49sr"]
Jul 29 04:40:30.449519 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:30.449488    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1fcfbd-695769d546-vpdkn"]
Jul 29 04:40:30.452777 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:30.452759    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1fcfbd-695769d546-vpdkn"
Jul 29 04:40:30.456009 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:30.455993    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-1fcfbd-dockercfg-njrx4\""
Jul 29 04:40:30.467337 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:30.467316    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1fcfbd-695769d546-vpdkn"]
Jul 29 04:40:30.561600 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:30.561566    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fcfbd-78c65659b8-jtr2j"
Jul 29 04:40:30.591993 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:30.591965    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm4gl\" (UniqueName: \"kubernetes.io/projected/b186e317-2df6-4f9b-a1d8-30fa5c8495e2-kube-api-access-dm4gl\") pod \"payload-pre-processing-e2e-inf-a-1fcfbd-c6985bd7b-d49sr\" (UID: \"b186e317-2df6-4f9b-a1d8-30fa5c8495e2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1fcfbd-c6985bd7b-d49sr"
Jul 29 04:40:30.592111 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:30.592004    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f3d6632-6e47-447c-81ef-ad97ea1538b8-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1fcfbd-695769d546-vpdkn\" (UID: \"3f3d6632-6e47-447c-81ef-ad97ea1538b8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1fcfbd-695769d546-vpdkn"
Jul 29 04:40:30.592164 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:30.592124    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b186e317-2df6-4f9b-a1d8-30fa5c8495e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1fcfbd-c6985bd7b-d49sr\" (UID: \"b186e317-2df6-4f9b-a1d8-30fa5c8495e2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1fcfbd-c6985bd7b-d49sr"
Jul 29 04:40:30.592164 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:30.592159    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlz4l\" (UniqueName: \"kubernetes.io/projected/3f3d6632-6e47-447c-81ef-ad97ea1538b8-kube-api-access-rlz4l\") pod \"payload-processing-e2e-inf-a-1fcfbd-695769d546-vpdkn\" (UID: \"3f3d6632-6e47-447c-81ef-ad97ea1538b8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1fcfbd-695769d546-vpdkn"
Jul 29 04:40:30.692888 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:30.692846    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b186e317-2df6-4f9b-a1d8-30fa5c8495e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1fcfbd-c6985bd7b-d49sr\" (UID: \"b186e317-2df6-4f9b-a1d8-30fa5c8495e2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1fcfbd-c6985bd7b-d49sr"
Jul 29 04:40:30.693344 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:30.692902    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlz4l\" (UniqueName: \"kubernetes.io/projected/3f3d6632-6e47-447c-81ef-ad97ea1538b8-kube-api-access-rlz4l\") pod \"payload-processing-e2e-inf-a-1fcfbd-695769d546-vpdkn\" (UID: \"3f3d6632-6e47-447c-81ef-ad97ea1538b8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1fcfbd-695769d546-vpdkn"
Jul 29 04:40:30.693344 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:30.692955    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm4gl\" (UniqueName: \"kubernetes.io/projected/b186e317-2df6-4f9b-a1d8-30fa5c8495e2-kube-api-access-dm4gl\") pod \"payload-pre-processing-e2e-inf-a-1fcfbd-c6985bd7b-d49sr\" (UID: \"b186e317-2df6-4f9b-a1d8-30fa5c8495e2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1fcfbd-c6985bd7b-d49sr"
Jul 29 04:40:30.693344 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:30.692985    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f3d6632-6e47-447c-81ef-ad97ea1538b8-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1fcfbd-695769d546-vpdkn\" (UID: \"3f3d6632-6e47-447c-81ef-ad97ea1538b8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1fcfbd-695769d546-vpdkn"
Jul 29 04:40:30.693734 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:30.693716    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b186e317-2df6-4f9b-a1d8-30fa5c8495e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1fcfbd-c6985bd7b-d49sr\" (UID: \"b186e317-2df6-4f9b-a1d8-30fa5c8495e2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1fcfbd-c6985bd7b-d49sr"
Jul 29 04:40:30.693777 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:30.693718    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f3d6632-6e47-447c-81ef-ad97ea1538b8-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1fcfbd-695769d546-vpdkn\" (UID: \"3f3d6632-6e47-447c-81ef-ad97ea1538b8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1fcfbd-695769d546-vpdkn"
Jul 29 04:40:30.697846 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:30.697810    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1fcfbd-78c65659b8-jtr2j"]
Jul 29 04:40:30.699122 ip-10-0-135-138 kubenswrapper[2570]: W0729 04:40:30.699097    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e6ccb39_07ea_45e1_ad39_5da90124c5ec.slice/crio-def86ac08fd97471e4d9f9df198e9a3606b11b00d9f052ad6dcc7e2d2c732ee6 WatchSource:0}: Error finding container def86ac08fd97471e4d9f9df198e9a3606b11b00d9f052ad6dcc7e2d2c732ee6: Status 404 returned error can't find the container with id def86ac08fd97471e4d9f9df198e9a3606b11b00d9f052ad6dcc7e2d2c732ee6
Jul 29 04:40:30.700574 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:30.700547    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 04:40:30.702144 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:30.702117    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm4gl\" (UniqueName: \"kubernetes.io/projected/b186e317-2df6-4f9b-a1d8-30fa5c8495e2-kube-api-access-dm4gl\") pod \"payload-pre-processing-e2e-inf-a-1fcfbd-c6985bd7b-d49sr\" (UID: \"b186e317-2df6-4f9b-a1d8-30fa5c8495e2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1fcfbd-c6985bd7b-d49sr"
Jul 29 04:40:30.702232 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:30.702153    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlz4l\" (UniqueName: \"kubernetes.io/projected/3f3d6632-6e47-447c-81ef-ad97ea1538b8-kube-api-access-rlz4l\") pod \"payload-processing-e2e-inf-a-1fcfbd-695769d546-vpdkn\" (UID: \"3f3d6632-6e47-447c-81ef-ad97ea1538b8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1fcfbd-695769d546-vpdkn"
Jul 29 04:40:30.717976 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:30.717956    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1fcfbd-c6985bd7b-d49sr"
Jul 29 04:40:30.762036 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:30.762011    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1fcfbd-695769d546-vpdkn"
Jul 29 04:40:30.888720 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:30.888689    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1fcfbd-c6985bd7b-d49sr"]
Jul 29 04:40:30.890420 ip-10-0-135-138 kubenswrapper[2570]: W0729 04:40:30.890389    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb186e317_2df6_4f9b_a1d8_30fa5c8495e2.slice/crio-81b7f7126e4fe788dbd93180a610deca56c9cf97d52708923113dd12e45b2e24 WatchSource:0}: Error finding container 81b7f7126e4fe788dbd93180a610deca56c9cf97d52708923113dd12e45b2e24: Status 404 returned error can't find the container with id 81b7f7126e4fe788dbd93180a610deca56c9cf97d52708923113dd12e45b2e24
Jul 29 04:40:30.930282 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:30.930260    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1fcfbd-695769d546-vpdkn"]
Jul 29 04:40:30.932954 ip-10-0-135-138 kubenswrapper[2570]: W0729 04:40:30.932929    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f3d6632_6e47_447c_81ef_ad97ea1538b8.slice/crio-e4fd7d9a182b59674a3c7310ca9911a998fdd94981ac9b5573376d4044e3defc WatchSource:0}: Error finding container e4fd7d9a182b59674a3c7310ca9911a998fdd94981ac9b5573376d4044e3defc: Status 404 returned error can't find the container with id e4fd7d9a182b59674a3c7310ca9911a998fdd94981ac9b5573376d4044e3defc
Jul 29 04:40:31.014493 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:31.014449    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1fcfbd-c6985bd7b-d49sr" event={"ID":"b186e317-2df6-4f9b-a1d8-30fa5c8495e2","Type":"ContainerStarted","Data":"255deca879fdd8ceed95ac7e00f0141370c72414ea7800523990f2cf9ca21611"}
Jul 29 04:40:31.014653 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:31.014505    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1fcfbd-c6985bd7b-d49sr" event={"ID":"b186e317-2df6-4f9b-a1d8-30fa5c8495e2","Type":"ContainerStarted","Data":"81b7f7126e4fe788dbd93180a610deca56c9cf97d52708923113dd12e45b2e24"}
Jul 29 04:40:31.014717 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:31.014653    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1fcfbd-c6985bd7b-d49sr"
Jul 29 04:40:31.016466 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:31.016436    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1fcfbd-695769d546-vpdkn" event={"ID":"3f3d6632-6e47-447c-81ef-ad97ea1538b8","Type":"ContainerStarted","Data":"2414b872965f80a84dbc4623041ffda6863b7e15acdbae344c174459a623018b"}
Jul 29 04:40:31.016595 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:31.016487    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1fcfbd-695769d546-vpdkn" event={"ID":"3f3d6632-6e47-447c-81ef-ad97ea1538b8","Type":"ContainerStarted","Data":"e4fd7d9a182b59674a3c7310ca9911a998fdd94981ac9b5573376d4044e3defc"}
Jul 29 04:40:31.016595 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:31.016560    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-1fcfbd-695769d546-vpdkn"
Jul 29 04:40:31.017586 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:31.017565    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fcfbd-78c65659b8-jtr2j" event={"ID":"7e6ccb39-07ea-45e1-ad39-5da90124c5ec","Type":"ContainerStarted","Data":"def86ac08fd97471e4d9f9df198e9a3606b11b00d9f052ad6dcc7e2d2c732ee6"}
Jul 29 04:40:31.033088 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:31.033048    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1fcfbd-c6985bd7b-d49sr" podStartSLOduration=1.033035822 podStartE2EDuration="1.033035822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:40:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:40:31.030345232 +0000 UTC m=+2695.760476322" watchObservedRunningTime="2026-07-29 04:40:31.033035822 +0000 UTC m=+2695.763166922"
Jul 29 04:40:31.048703 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:31.048660    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-1fcfbd-695769d546-vpdkn" podStartSLOduration=1.048648139 podStartE2EDuration="1.048648139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:40:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:40:31.046649609 +0000 UTC m=+2695.776780700" watchObservedRunningTime="2026-07-29 04:40:31.048648139 +0000 UTC m=+2695.778779208"
Jul 29 04:40:32.025781 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:32.025745    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fcfbd-78c65659b8-jtr2j" event={"ID":"7e6ccb39-07ea-45e1-ad39-5da90124c5ec","Type":"ContainerStarted","Data":"a99d0dd6501f2620a200173a1f02faf1074749532639a011533843edffff14b5"}
Jul 29 04:40:32.026222 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:32.026200    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fcfbd-78c65659b8-jtr2j"
Jul 29 04:40:32.044236 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:32.044180    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fcfbd-78c65659b8-jtr2j" podStartSLOduration=2.56273939 podStartE2EDuration="3.044164559s" totalImagesPullingTime="481.425169ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:40:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:40:32.043094515 +0000 UTC m=+2696.773225607" watchObservedRunningTime="2026-07-29 04:40:32.044164559 +0000 UTC m=+2696.774295628"
Jul 29 04:40:36.062801 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:36.062766    2570 scope.go:117] "RemoveContainer" containerID="6644b716e4429a8c8cffe521f7b89f8d6c3fc635767403b2618e02823615f407"
Jul 29 04:40:39.046487 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:39.046453    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fcfbd-78c65659b8-jtr2j"
Jul 29 04:40:42.027895 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:42.027866    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1fcfbd-c6985bd7b-d49sr"
Jul 29 04:40:42.028350 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:42.027946    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-1fcfbd-695769d546-vpdkn"
Jul 29 04:40:58.430573 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:58.430536    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a32259-cdbb84d64-r2qsw"]
Jul 29 04:40:58.434339 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:58.434318    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a32259-cdbb84d64-r2qsw"
Jul 29 04:40:58.436993 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:58.436974    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-a32259\""
Jul 29 04:40:58.447355 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:58.447330    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a32259-cdbb84d64-r2qsw"]
Jul 29 04:40:58.477001 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:58.476970    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a32259-7b5c7d496d-7dd6f"]
Jul 29 04:40:58.481166 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:58.481146    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a32259-7b5c7d496d-7dd6f"
Jul 29 04:40:58.484106 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:58.484087    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-a32259-dockercfg-vm9gc\""
Jul 29 04:40:58.488717 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:58.488695    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a32259-7b5c7d496d-7dd6f"]
Jul 29 04:40:58.549311 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:58.549284    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4jv7\" (UniqueName: \"kubernetes.io/projected/fe6ee826-e478-46a5-8e17-02d4e608755d-kube-api-access-r4jv7\") pod \"payload-pre-processing-e2e-inf-b-a32259-cdbb84d64-r2qsw\" (UID: \"fe6ee826-e478-46a5-8e17-02d4e608755d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a32259-cdbb84d64-r2qsw"
Jul 29 04:40:58.549447 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:58.549363    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe6ee826-e478-46a5-8e17-02d4e608755d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-a32259-cdbb84d64-r2qsw\" (UID: \"fe6ee826-e478-46a5-8e17-02d4e608755d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a32259-cdbb84d64-r2qsw"
Jul 29 04:40:58.549447 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:58.549386    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7027dd4b-6c64-40c5-b279-3fd8b9937788-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-a32259-7b5c7d496d-7dd6f\" (UID: \"7027dd4b-6c64-40c5-b279-3fd8b9937788\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a32259-7b5c7d496d-7dd6f"
Jul 29 04:40:58.549447 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:58.549400    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrfhc\" (UniqueName: \"kubernetes.io/projected/7027dd4b-6c64-40c5-b279-3fd8b9937788-kube-api-access-vrfhc\") pod \"payload-processing-e2e-inf-b-a32259-7b5c7d496d-7dd6f\" (UID: \"7027dd4b-6c64-40c5-b279-3fd8b9937788\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a32259-7b5c7d496d-7dd6f"
Jul 29 04:40:58.650159 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:58.650123    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4jv7\" (UniqueName: \"kubernetes.io/projected/fe6ee826-e478-46a5-8e17-02d4e608755d-kube-api-access-r4jv7\") pod \"payload-pre-processing-e2e-inf-b-a32259-cdbb84d64-r2qsw\" (UID: \"fe6ee826-e478-46a5-8e17-02d4e608755d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a32259-cdbb84d64-r2qsw"
Jul 29 04:40:58.650333 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:58.650206    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe6ee826-e478-46a5-8e17-02d4e608755d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-a32259-cdbb84d64-r2qsw\" (UID: \"fe6ee826-e478-46a5-8e17-02d4e608755d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a32259-cdbb84d64-r2qsw"
Jul 29 04:40:58.650333 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:58.650227    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7027dd4b-6c64-40c5-b279-3fd8b9937788-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-a32259-7b5c7d496d-7dd6f\" (UID: \"7027dd4b-6c64-40c5-b279-3fd8b9937788\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a32259-7b5c7d496d-7dd6f"
Jul 29 04:40:58.650333 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:58.650244    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrfhc\" (UniqueName: \"kubernetes.io/projected/7027dd4b-6c64-40c5-b279-3fd8b9937788-kube-api-access-vrfhc\") pod \"payload-processing-e2e-inf-b-a32259-7b5c7d496d-7dd6f\" (UID: \"7027dd4b-6c64-40c5-b279-3fd8b9937788\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a32259-7b5c7d496d-7dd6f"
Jul 29 04:40:58.650951 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:58.650928    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe6ee826-e478-46a5-8e17-02d4e608755d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-a32259-cdbb84d64-r2qsw\" (UID: \"fe6ee826-e478-46a5-8e17-02d4e608755d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a32259-cdbb84d64-r2qsw"
Jul 29 04:40:58.650951 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:58.650938    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7027dd4b-6c64-40c5-b279-3fd8b9937788-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-a32259-7b5c7d496d-7dd6f\" (UID: \"7027dd4b-6c64-40c5-b279-3fd8b9937788\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a32259-7b5c7d496d-7dd6f"
Jul 29 04:40:58.658072 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:58.658051    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrfhc\" (UniqueName: \"kubernetes.io/projected/7027dd4b-6c64-40c5-b279-3fd8b9937788-kube-api-access-vrfhc\") pod \"payload-processing-e2e-inf-b-a32259-7b5c7d496d-7dd6f\" (UID: \"7027dd4b-6c64-40c5-b279-3fd8b9937788\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a32259-7b5c7d496d-7dd6f"
Jul 29 04:40:58.658154 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:58.658109    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4jv7\" (UniqueName: \"kubernetes.io/projected/fe6ee826-e478-46a5-8e17-02d4e608755d-kube-api-access-r4jv7\") pod \"payload-pre-processing-e2e-inf-b-a32259-cdbb84d64-r2qsw\" (UID: \"fe6ee826-e478-46a5-8e17-02d4e608755d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a32259-cdbb84d64-r2qsw"
Jul 29 04:40:58.745751 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:58.745683    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a32259-cdbb84d64-r2qsw"
Jul 29 04:40:58.792693 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:58.792666    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a32259-7b5c7d496d-7dd6f"
Jul 29 04:40:58.897346 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:58.897317    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a32259-cdbb84d64-r2qsw"]
Jul 29 04:40:58.899793 ip-10-0-135-138 kubenswrapper[2570]: W0729 04:40:58.899740    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe6ee826_e478_46a5_8e17_02d4e608755d.slice/crio-4a88087adea7f1d710a82dbc84efd1172ebfd810a8a813d5ab969928836a020f WatchSource:0}: Error finding container 4a88087adea7f1d710a82dbc84efd1172ebfd810a8a813d5ab969928836a020f: Status 404 returned error can't find the container with id 4a88087adea7f1d710a82dbc84efd1172ebfd810a8a813d5ab969928836a020f
Jul 29 04:40:58.952813 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:58.952665    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a32259-7b5c7d496d-7dd6f"]
Jul 29 04:40:58.954704 ip-10-0-135-138 kubenswrapper[2570]: W0729 04:40:58.954681    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7027dd4b_6c64_40c5_b279_3fd8b9937788.slice/crio-af8419c8a07ee1cecf155b115750c38a5e160494cfb825c0abdd79fde73e15a6 WatchSource:0}: Error finding container af8419c8a07ee1cecf155b115750c38a5e160494cfb825c0abdd79fde73e15a6: Status 404 returned error can't find the container with id af8419c8a07ee1cecf155b115750c38a5e160494cfb825c0abdd79fde73e15a6
Jul 29 04:40:59.160966 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:59.160930    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a32259-7b5c7d496d-7dd6f" event={"ID":"7027dd4b-6c64-40c5-b279-3fd8b9937788","Type":"ContainerStarted","Data":"94b616809ed3b4ad41163c265a27d880be6a2e2c40ebc3041e906ddc81e1105c"}
Jul 29 04:40:59.161142 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:59.160987    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a32259-7b5c7d496d-7dd6f" event={"ID":"7027dd4b-6c64-40c5-b279-3fd8b9937788","Type":"ContainerStarted","Data":"af8419c8a07ee1cecf155b115750c38a5e160494cfb825c0abdd79fde73e15a6"}
Jul 29 04:40:59.161142 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:59.161032    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-a32259-7b5c7d496d-7dd6f"
Jul 29 04:40:59.162404 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:59.162380    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a32259-cdbb84d64-r2qsw" event={"ID":"fe6ee826-e478-46a5-8e17-02d4e608755d","Type":"ContainerStarted","Data":"a30df8533e53746c31f2ed9a62fff755bdbd4c5e13535e00e72bcfe628d358d8"}
Jul 29 04:40:59.162531 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:59.162423    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a32259-cdbb84d64-r2qsw" event={"ID":"fe6ee826-e478-46a5-8e17-02d4e608755d","Type":"ContainerStarted","Data":"4a88087adea7f1d710a82dbc84efd1172ebfd810a8a813d5ab969928836a020f"}
Jul 29 04:40:59.162531 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:59.162458    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a32259-cdbb84d64-r2qsw"
Jul 29 04:40:59.179534 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:59.179482    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-a32259-7b5c7d496d-7dd6f" podStartSLOduration=1.179469657 podStartE2EDuration="1.179469657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:40:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:40:59.178173575 +0000 UTC m=+2723.908304658" watchObservedRunningTime="2026-07-29 04:40:59.179469657 +0000 UTC m=+2723.909600717"
Jul 29 04:40:59.196996 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:40:59.196955    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a32259-cdbb84d64-r2qsw" podStartSLOduration=1.196941418 podStartE2EDuration="1.196941418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:40:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:40:59.195922391 +0000 UTC m=+2723.926053463" watchObservedRunningTime="2026-07-29 04:40:59.196941418 +0000 UTC m=+2723.927072488"
Jul 29 04:41:10.169568 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:41:10.169538    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-a32259-7b5c7d496d-7dd6f"
Jul 29 04:41:10.170075 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:41:10.169607    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a32259-cdbb84d64-r2qsw"
Jul 29 04:41:53.932344 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:41:53.932305    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-a32259/test-model-a32259-kserve-6f548cd569-gkvkc"]
Jul 29 04:41:53.936286 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:41:53.936268    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a32259/test-model-a32259-kserve-6f548cd569-gkvkc"
Jul 29 04:41:53.938900 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:41:53.938880    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-a32259\"/\"openshift-service-ca.crt\""
Jul 29 04:41:53.940854 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:41:53.940675    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-a32259\"/\"test-model-a32259-kserve-self-signed-certs\""
Jul 29 04:41:53.940854 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:41:53.940732    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-a32259\"/\"kube-root-ca.crt\""
Jul 29 04:41:53.941060 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:41:53.940966    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-a32259\"/\"default-dockercfg-96c7q\""
Jul 29 04:41:53.943779 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:41:53.943759    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-a32259/test-model-a32259-kserve-6f548cd569-gkvkc"]
Jul 29 04:41:54.061280 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:41:54.061252    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54c60920-bc3a-4cee-98bb-ec589ae2ac7c-tmp-dir\") pod \"test-model-a32259-kserve-6f548cd569-gkvkc\" (UID: \"54c60920-bc3a-4cee-98bb-ec589ae2ac7c\") " pod="ai-tenant-e2e-inf-b-a32259/test-model-a32259-kserve-6f548cd569-gkvkc"
Jul 29 04:41:54.061438 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:41:54.061301    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54c60920-bc3a-4cee-98bb-ec589ae2ac7c-model-cache\") pod \"test-model-a32259-kserve-6f548cd569-gkvkc\" (UID: \"54c60920-bc3a-4cee-98bb-ec589ae2ac7c\") " pod="ai-tenant-e2e-inf-b-a32259/test-model-a32259-kserve-6f548cd569-gkvkc"
Jul 29 04:41:54.061438 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:41:54.061351    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54c60920-bc3a-4cee-98bb-ec589ae2ac7c-home\") pod \"test-model-a32259-kserve-6f548cd569-gkvkc\" (UID: \"54c60920-bc3a-4cee-98bb-ec589ae2ac7c\") " pod="ai-tenant-e2e-inf-b-a32259/test-model-a32259-kserve-6f548cd569-gkvkc"
Jul 29 04:41:54.061438 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:41:54.061419    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54c60920-bc3a-4cee-98bb-ec589ae2ac7c-tls-certs\") pod \"test-model-a32259-kserve-6f548cd569-gkvkc\" (UID: \"54c60920-bc3a-4cee-98bb-ec589ae2ac7c\") " pod="ai-tenant-e2e-inf-b-a32259/test-model-a32259-kserve-6f548cd569-gkvkc"
Jul 29 04:41:54.061584 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:41:54.061440    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9xvq\" (UniqueName: \"kubernetes.io/projected/54c60920-bc3a-4cee-98bb-ec589ae2ac7c-kube-api-access-d9xvq\") pod \"test-model-a32259-kserve-6f548cd569-gkvkc\" (UID: \"54c60920-bc3a-4cee-98bb-ec589ae2ac7c\") " pod="ai-tenant-e2e-inf-b-a32259/test-model-a32259-kserve-6f548cd569-gkvkc"
Jul 29 04:41:54.061584 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:41:54.061496    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54c60920-bc3a-4cee-98bb-ec589ae2ac7c-dshm\") pod \"test-model-a32259-kserve-6f548cd569-gkvkc\" (UID: \"54c60920-bc3a-4cee-98bb-ec589ae2ac7c\") " pod="ai-tenant-e2e-inf-b-a32259/test-model-a32259-kserve-6f548cd569-gkvkc"
Jul 29 04:41:54.162452 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:41:54.162410    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54c60920-bc3a-4cee-98bb-ec589ae2ac7c-tmp-dir\") pod \"test-model-a32259-kserve-6f548cd569-gkvkc\" (UID: \"54c60920-bc3a-4cee-98bb-ec589ae2ac7c\") " pod="ai-tenant-e2e-inf-b-a32259/test-model-a32259-kserve-6f548cd569-gkvkc"
Jul 29 04:41:54.162652 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:41:54.162477    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54c60920-bc3a-4cee-98bb-ec589ae2ac7c-model-cache\") pod \"test-model-a32259-kserve-6f548cd569-gkvkc\" (UID: \"54c60920-bc3a-4cee-98bb-ec589ae2ac7c\") " pod="ai-tenant-e2e-inf-b-a32259/test-model-a32259-kserve-6f548cd569-gkvkc"
Jul 29 04:41:54.162652 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:41:54.162505    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54c60920-bc3a-4cee-98bb-ec589ae2ac7c-home\") pod \"test-model-a32259-kserve-6f548cd569-gkvkc\" (UID: \"54c60920-bc3a-4cee-98bb-ec589ae2ac7c\") " pod="ai-tenant-e2e-inf-b-a32259/test-model-a32259-kserve-6f548cd569-gkvkc"
Jul 29 04:41:54.162652 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:41:54.162567    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54c60920-bc3a-4cee-98bb-ec589ae2ac7c-tls-certs\") pod \"test-model-a32259-kserve-6f548cd569-gkvkc\" (UID: \"54c60920-bc3a-4cee-98bb-ec589ae2ac7c\") " pod="ai-tenant-e2e-inf-b-a32259/test-model-a32259-kserve-6f548cd569-gkvkc"
Jul 29 04:41:54.162652 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:41:54.162591    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9xvq\" (UniqueName: \"kubernetes.io/projected/54c60920-bc3a-4cee-98bb-ec589ae2ac7c-kube-api-access-d9xvq\") pod \"test-model-a32259-kserve-6f548cd569-gkvkc\" (UID: \"54c60920-bc3a-4cee-98bb-ec589ae2ac7c\") " pod="ai-tenant-e2e-inf-b-a32259/test-model-a32259-kserve-6f548cd569-gkvkc"
Jul 29 04:41:54.162652 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:41:54.162630    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54c60920-bc3a-4cee-98bb-ec589ae2ac7c-dshm\") pod \"test-model-a32259-kserve-6f548cd569-gkvkc\" (UID: \"54c60920-bc3a-4cee-98bb-ec589ae2ac7c\") " pod="ai-tenant-e2e-inf-b-a32259/test-model-a32259-kserve-6f548cd569-gkvkc"
Jul 29 04:41:54.162961 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:41:54.162908    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54c60920-bc3a-4cee-98bb-ec589ae2ac7c-tmp-dir\") pod \"test-model-a32259-kserve-6f548cd569-gkvkc\" (UID: \"54c60920-bc3a-4cee-98bb-ec589ae2ac7c\") " pod="ai-tenant-e2e-inf-b-a32259/test-model-a32259-kserve-6f548cd569-gkvkc"
Jul 29 04:41:54.162961 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:41:54.162923    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54c60920-bc3a-4cee-98bb-ec589ae2ac7c-model-cache\") pod \"test-model-a32259-kserve-6f548cd569-gkvkc\" (UID: \"54c60920-bc3a-4cee-98bb-ec589ae2ac7c\") " pod="ai-tenant-e2e-inf-b-a32259/test-model-a32259-kserve-6f548cd569-gkvkc"
Jul 29 04:41:54.163063 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:41:54.162960    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54c60920-bc3a-4cee-98bb-ec589ae2ac7c-home\") pod \"test-model-a32259-kserve-6f548cd569-gkvkc\" (UID: \"54c60920-bc3a-4cee-98bb-ec589ae2ac7c\") " pod="ai-tenant-e2e-inf-b-a32259/test-model-a32259-kserve-6f548cd569-gkvkc"
Jul 29 04:41:54.165086 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:41:54.165066    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54c60920-bc3a-4cee-98bb-ec589ae2ac7c-dshm\") pod \"test-model-a32259-kserve-6f548cd569-gkvkc\" (UID: \"54c60920-bc3a-4cee-98bb-ec589ae2ac7c\") " pod="ai-tenant-e2e-inf-b-a32259/test-model-a32259-kserve-6f548cd569-gkvkc"
Jul 29 04:41:54.165376 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:41:54.165357    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54c60920-bc3a-4cee-98bb-ec589ae2ac7c-tls-certs\") pod \"test-model-a32259-kserve-6f548cd569-gkvkc\" (UID: \"54c60920-bc3a-4cee-98bb-ec589ae2ac7c\") " pod="ai-tenant-e2e-inf-b-a32259/test-model-a32259-kserve-6f548cd569-gkvkc"
Jul 29 04:41:54.170481 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:41:54.170458    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9xvq\" (UniqueName: \"kubernetes.io/projected/54c60920-bc3a-4cee-98bb-ec589ae2ac7c-kube-api-access-d9xvq\") pod \"test-model-a32259-kserve-6f548cd569-gkvkc\" (UID: \"54c60920-bc3a-4cee-98bb-ec589ae2ac7c\") " pod="ai-tenant-e2e-inf-b-a32259/test-model-a32259-kserve-6f548cd569-gkvkc"
Jul 29 04:41:54.248473 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:41:54.248398    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a32259/test-model-a32259-kserve-6f548cd569-gkvkc"
Jul 29 04:41:54.383948 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:41:54.383922    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-a32259/test-model-a32259-kserve-6f548cd569-gkvkc"]
Jul 29 04:41:54.385964 ip-10-0-135-138 kubenswrapper[2570]: W0729 04:41:54.385936    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod54c60920_bc3a_4cee_98bb_ec589ae2ac7c.slice/crio-7882b4812ae45ec6f0c25a0be74b6b577f4b30335aff16a955fe16384a576c92 WatchSource:0}: Error finding container 7882b4812ae45ec6f0c25a0be74b6b577f4b30335aff16a955fe16384a576c92: Status 404 returned error can't find the container with id 7882b4812ae45ec6f0c25a0be74b6b577f4b30335aff16a955fe16384a576c92
Jul 29 04:41:54.446414 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:41:54.446381    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a32259/test-model-a32259-kserve-6f548cd569-gkvkc" event={"ID":"54c60920-bc3a-4cee-98bb-ec589ae2ac7c","Type":"ContainerStarted","Data":"387b5249c4b47e8e40eb6ad12a480ef3f01a65dc9164372598accea71edf82a0"}
Jul 29 04:41:54.446562 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:41:54.446428    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a32259/test-model-a32259-kserve-6f548cd569-gkvkc" event={"ID":"54c60920-bc3a-4cee-98bb-ec589ae2ac7c","Type":"ContainerStarted","Data":"7882b4812ae45ec6f0c25a0be74b6b577f4b30335aff16a955fe16384a576c92"}
Jul 29 04:41:54.461510 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:41:54.461316    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-a32259/test-model-a32259-kserve-6f548cd569-gkvkc" podStartSLOduration=1.46129693 podStartE2EDuration="1.46129693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:41:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:41:54.460895017 +0000 UTC m=+2779.191026088" watchObservedRunningTime="2026-07-29 04:41:54.46129693 +0000 UTC m=+2779.191428000"
Jul 29 04:42:04.249282 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:04.249208    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-a32259/test-model-a32259-kserve-6f548cd569-gkvkc"
Jul 29 04:42:04.249282 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:04.249257    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-a32259/test-model-a32259-kserve-6f548cd569-gkvkc"
Jul 29 04:42:04.262083 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:04.262059    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-a32259/test-model-a32259-kserve-6f548cd569-gkvkc"
Jul 29 04:42:04.509297 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:04.509217    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-a32259/test-model-a32259-kserve-6f548cd569-gkvkc"
Jul 29 04:42:33.030733 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:33.030687    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-a32259/test-model-a32259-kserve-6f548cd569-gkvkc"]
Jul 29 04:42:43.635285 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:43.635251    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1fcfbd-78c65659b8-jtr2j"]
Jul 29 04:42:43.635807 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:43.635485    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fcfbd-78c65659b8-jtr2j" podUID="7e6ccb39-07ea-45e1-ad39-5da90124c5ec" containerName="maas-api" containerID="cri-o://a99d0dd6501f2620a200173a1f02faf1074749532639a011533843edffff14b5" gracePeriod=30
Jul 29 04:42:43.677105 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:43.677076    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1fcfbd-695769d546-vpdkn"]
Jul 29 04:42:43.677356 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:43.677311    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-1fcfbd-695769d546-vpdkn" podUID="3f3d6632-6e47-447c-81ef-ad97ea1538b8" containerName="payload-processing" containerID="cri-o://2414b872965f80a84dbc4623041ffda6863b7e15acdbae344c174459a623018b" gracePeriod=30
Jul 29 04:42:43.685215 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:43.685189    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1fcfbd-c6985bd7b-d49sr"]
Jul 29 04:42:43.685448 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:43.685405    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1fcfbd-c6985bd7b-d49sr" podUID="b186e317-2df6-4f9b-a1d8-30fa5c8495e2" containerName="payload-pre-processing" containerID="cri-o://255deca879fdd8ceed95ac7e00f0141370c72414ea7800523990f2cf9ca21611" gracePeriod=30
Jul 29 04:42:43.955019 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:43.954988    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1fcfbd-695769d546-vpdkn"
Jul 29 04:42:43.991846 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:43.991805    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fcfbd-78c65659b8-jtr2j"
Jul 29 04:42:43.995098 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:43.995080    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1fcfbd-c6985bd7b-d49sr"
Jul 29 04:42:44.108873 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:44.108804    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e6ccb39-07ea-45e1-ad39-5da90124c5ec-maas-api-tls\") pod \"7e6ccb39-07ea-45e1-ad39-5da90124c5ec\" (UID: \"7e6ccb39-07ea-45e1-ad39-5da90124c5ec\") "
Jul 29 04:42:44.109078 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:44.108893    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e6ccb39-07ea-45e1-ad39-5da90124c5ec-openshift-service-ca\") pod \"7e6ccb39-07ea-45e1-ad39-5da90124c5ec\" (UID: \"7e6ccb39-07ea-45e1-ad39-5da90124c5ec\") "
Jul 29 04:42:44.109078 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:44.108959    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b186e317-2df6-4f9b-a1d8-30fa5c8495e2-plugins-config-volume\") pod \"b186e317-2df6-4f9b-a1d8-30fa5c8495e2\" (UID: \"b186e317-2df6-4f9b-a1d8-30fa5c8495e2\") "
Jul 29 04:42:44.109078 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:44.109043    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f3d6632-6e47-447c-81ef-ad97ea1538b8-plugins-config-volume\") pod \"3f3d6632-6e47-447c-81ef-ad97ea1538b8\" (UID: \"3f3d6632-6e47-447c-81ef-ad97ea1538b8\") "
Jul 29 04:42:44.109078 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:44.109078    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xhjjl\" (UniqueName: \"kubernetes.io/projected/7e6ccb39-07ea-45e1-ad39-5da90124c5ec-kube-api-access-xhjjl\") pod \"7e6ccb39-07ea-45e1-ad39-5da90124c5ec\" (UID: \"7e6ccb39-07ea-45e1-ad39-5da90124c5ec\") "
Jul 29 04:42:44.109340 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:44.109102    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rlz4l\" (UniqueName: \"kubernetes.io/projected/3f3d6632-6e47-447c-81ef-ad97ea1538b8-kube-api-access-rlz4l\") pod \"3f3d6632-6e47-447c-81ef-ad97ea1538b8\" (UID: \"3f3d6632-6e47-447c-81ef-ad97ea1538b8\") "
Jul 29 04:42:44.109340 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:44.109233    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dm4gl\" (UniqueName: \"kubernetes.io/projected/b186e317-2df6-4f9b-a1d8-30fa5c8495e2-kube-api-access-dm4gl\") pod \"b186e317-2df6-4f9b-a1d8-30fa5c8495e2\" (UID: \"b186e317-2df6-4f9b-a1d8-30fa5c8495e2\") "
Jul 29 04:42:44.109463 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:44.109439    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f3d6632-6e47-447c-81ef-ad97ea1538b8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3f3d6632-6e47-447c-81ef-ad97ea1538b8" (UID: "3f3d6632-6e47-447c-81ef-ad97ea1538b8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 04:42:44.109518 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:44.109464    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e6ccb39-07ea-45e1-ad39-5da90124c5ec-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7e6ccb39-07ea-45e1-ad39-5da90124c5ec" (UID: "7e6ccb39-07ea-45e1-ad39-5da90124c5ec"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 04:42:44.109735 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:44.109711    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b186e317-2df6-4f9b-a1d8-30fa5c8495e2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b186e317-2df6-4f9b-a1d8-30fa5c8495e2" (UID: "b186e317-2df6-4f9b-a1d8-30fa5c8495e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 04:42:44.109933 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:44.109876    2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e6ccb39-07ea-45e1-ad39-5da90124c5ec-openshift-service-ca\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:42:44.109933 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:44.109897    2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b186e317-2df6-4f9b-a1d8-30fa5c8495e2-plugins-config-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:42:44.109933 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:44.109910    2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f3d6632-6e47-447c-81ef-ad97ea1538b8-plugins-config-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:42:44.111614 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:44.111586    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b186e317-2df6-4f9b-a1d8-30fa5c8495e2-kube-api-access-dm4gl" (OuterVolumeSpecName: "kube-api-access-dm4gl") pod "b186e317-2df6-4f9b-a1d8-30fa5c8495e2" (UID: "b186e317-2df6-4f9b-a1d8-30fa5c8495e2"). InnerVolumeSpecName "kube-api-access-dm4gl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 04:42:44.111724 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:44.111679    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e6ccb39-07ea-45e1-ad39-5da90124c5ec-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7e6ccb39-07ea-45e1-ad39-5da90124c5ec" (UID: "7e6ccb39-07ea-45e1-ad39-5da90124c5ec"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 04:42:44.112181 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:44.112164    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e6ccb39-07ea-45e1-ad39-5da90124c5ec-kube-api-access-xhjjl" (OuterVolumeSpecName: "kube-api-access-xhjjl") pod "7e6ccb39-07ea-45e1-ad39-5da90124c5ec" (UID: "7e6ccb39-07ea-45e1-ad39-5da90124c5ec"). InnerVolumeSpecName "kube-api-access-xhjjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 04:42:44.112235 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:44.112177    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f3d6632-6e47-447c-81ef-ad97ea1538b8-kube-api-access-rlz4l" (OuterVolumeSpecName: "kube-api-access-rlz4l") pod "3f3d6632-6e47-447c-81ef-ad97ea1538b8" (UID: "3f3d6632-6e47-447c-81ef-ad97ea1538b8"). InnerVolumeSpecName "kube-api-access-rlz4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 04:42:44.211166 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:44.211099    2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e6ccb39-07ea-45e1-ad39-5da90124c5ec-maas-api-tls\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:42:44.211166 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:44.211123    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhjjl\" (UniqueName: \"kubernetes.io/projected/7e6ccb39-07ea-45e1-ad39-5da90124c5ec-kube-api-access-xhjjl\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:42:44.211166 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:44.211137    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlz4l\" (UniqueName: \"kubernetes.io/projected/3f3d6632-6e47-447c-81ef-ad97ea1538b8-kube-api-access-rlz4l\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:42:44.211166 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:44.211150    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dm4gl\" (UniqueName: \"kubernetes.io/projected/b186e317-2df6-4f9b-a1d8-30fa5c8495e2-kube-api-access-dm4gl\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:42:44.706832 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:44.706782    2570 generic.go:356] "Generic (PLEG): container finished" podID="3f3d6632-6e47-447c-81ef-ad97ea1538b8" containerID="2414b872965f80a84dbc4623041ffda6863b7e15acdbae344c174459a623018b" exitCode=0
Jul 29 04:42:44.707256 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:44.706843    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1fcfbd-695769d546-vpdkn"
Jul 29 04:42:44.707256 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:44.706869    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1fcfbd-695769d546-vpdkn" event={"ID":"3f3d6632-6e47-447c-81ef-ad97ea1538b8","Type":"ContainerDied","Data":"2414b872965f80a84dbc4623041ffda6863b7e15acdbae344c174459a623018b"}
Jul 29 04:42:44.707256 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:44.706938    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1fcfbd-695769d546-vpdkn" event={"ID":"3f3d6632-6e47-447c-81ef-ad97ea1538b8","Type":"ContainerDied","Data":"e4fd7d9a182b59674a3c7310ca9911a998fdd94981ac9b5573376d4044e3defc"}
Jul 29 04:42:44.707256 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:44.706972    2570 scope.go:117] "RemoveContainer" containerID="2414b872965f80a84dbc4623041ffda6863b7e15acdbae344c174459a623018b"
Jul 29 04:42:44.708957 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:44.708852    2570 generic.go:356] "Generic (PLEG): container finished" podID="7e6ccb39-07ea-45e1-ad39-5da90124c5ec" containerID="a99d0dd6501f2620a200173a1f02faf1074749532639a011533843edffff14b5" exitCode=0
Jul 29 04:42:44.708957 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:44.708906    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fcfbd-78c65659b8-jtr2j" event={"ID":"7e6ccb39-07ea-45e1-ad39-5da90124c5ec","Type":"ContainerDied","Data":"a99d0dd6501f2620a200173a1f02faf1074749532639a011533843edffff14b5"}
Jul 29 04:42:44.709147 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:44.708952    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fcfbd-78c65659b8-jtr2j" event={"ID":"7e6ccb39-07ea-45e1-ad39-5da90124c5ec","Type":"ContainerDied","Data":"def86ac08fd97471e4d9f9df198e9a3606b11b00d9f052ad6dcc7e2d2c732ee6"}
Jul 29 04:42:44.709147 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:44.709057    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fcfbd-78c65659b8-jtr2j"
Jul 29 04:42:44.710780 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:44.710731    2570 generic.go:356] "Generic (PLEG): container finished" podID="b186e317-2df6-4f9b-a1d8-30fa5c8495e2" containerID="255deca879fdd8ceed95ac7e00f0141370c72414ea7800523990f2cf9ca21611" exitCode=0
Jul 29 04:42:44.710932 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:44.710782    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1fcfbd-c6985bd7b-d49sr" event={"ID":"b186e317-2df6-4f9b-a1d8-30fa5c8495e2","Type":"ContainerDied","Data":"255deca879fdd8ceed95ac7e00f0141370c72414ea7800523990f2cf9ca21611"}
Jul 29 04:42:44.710932 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:44.710860    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1fcfbd-c6985bd7b-d49sr" event={"ID":"b186e317-2df6-4f9b-a1d8-30fa5c8495e2","Type":"ContainerDied","Data":"81b7f7126e4fe788dbd93180a610deca56c9cf97d52708923113dd12e45b2e24"}
Jul 29 04:42:44.710932 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:44.710764    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1fcfbd-c6985bd7b-d49sr"
Jul 29 04:42:44.719544 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:44.719523    2570 scope.go:117] "RemoveContainer" containerID="2414b872965f80a84dbc4623041ffda6863b7e15acdbae344c174459a623018b"
Jul 29 04:42:44.719880 ip-10-0-135-138 kubenswrapper[2570]: E0729 04:42:44.719858    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2414b872965f80a84dbc4623041ffda6863b7e15acdbae344c174459a623018b\": container with ID starting with 2414b872965f80a84dbc4623041ffda6863b7e15acdbae344c174459a623018b not found: ID does not exist" containerID="2414b872965f80a84dbc4623041ffda6863b7e15acdbae344c174459a623018b"
Jul 29 04:42:44.719973 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:44.719889    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2414b872965f80a84dbc4623041ffda6863b7e15acdbae344c174459a623018b"} err="failed to get container status \"2414b872965f80a84dbc4623041ffda6863b7e15acdbae344c174459a623018b\": rpc error: code = NotFound desc = could not find container \"2414b872965f80a84dbc4623041ffda6863b7e15acdbae344c174459a623018b\": container with ID starting with 2414b872965f80a84dbc4623041ffda6863b7e15acdbae344c174459a623018b not found: ID does not exist"
Jul 29 04:42:44.719973 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:44.719909    2570 scope.go:117] "RemoveContainer" containerID="a99d0dd6501f2620a200173a1f02faf1074749532639a011533843edffff14b5"
Jul 29 04:42:44.728281 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:44.728266    2570 scope.go:117] "RemoveContainer" containerID="a99d0dd6501f2620a200173a1f02faf1074749532639a011533843edffff14b5"
Jul 29 04:42:44.728531 ip-10-0-135-138 kubenswrapper[2570]: E0729 04:42:44.728512    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a99d0dd6501f2620a200173a1f02faf1074749532639a011533843edffff14b5\": container with ID starting with a99d0dd6501f2620a200173a1f02faf1074749532639a011533843edffff14b5 not found: ID does not exist" containerID="a99d0dd6501f2620a200173a1f02faf1074749532639a011533843edffff14b5"
Jul 29 04:42:44.728581 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:44.728536    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a99d0dd6501f2620a200173a1f02faf1074749532639a011533843edffff14b5"} err="failed to get container status \"a99d0dd6501f2620a200173a1f02faf1074749532639a011533843edffff14b5\": rpc error: code = NotFound desc = could not find container \"a99d0dd6501f2620a200173a1f02faf1074749532639a011533843edffff14b5\": container with ID starting with a99d0dd6501f2620a200173a1f02faf1074749532639a011533843edffff14b5 not found: ID does not exist"
Jul 29 04:42:44.728581 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:44.728554    2570 scope.go:117] "RemoveContainer" containerID="255deca879fdd8ceed95ac7e00f0141370c72414ea7800523990f2cf9ca21611"
Jul 29 04:42:44.734157 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:44.734137    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1fcfbd-695769d546-vpdkn"]
Jul 29 04:42:44.736899 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:44.736881    2570 scope.go:117] "RemoveContainer" containerID="255deca879fdd8ceed95ac7e00f0141370c72414ea7800523990f2cf9ca21611"
Jul 29 04:42:44.737168 ip-10-0-135-138 kubenswrapper[2570]: E0729 04:42:44.737150    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"255deca879fdd8ceed95ac7e00f0141370c72414ea7800523990f2cf9ca21611\": container with ID starting with 255deca879fdd8ceed95ac7e00f0141370c72414ea7800523990f2cf9ca21611 not found: ID does not exist" containerID="255deca879fdd8ceed95ac7e00f0141370c72414ea7800523990f2cf9ca21611"
Jul 29 04:42:44.737220 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:44.737176    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"255deca879fdd8ceed95ac7e00f0141370c72414ea7800523990f2cf9ca21611"} err="failed to get container status \"255deca879fdd8ceed95ac7e00f0141370c72414ea7800523990f2cf9ca21611\": rpc error: code = NotFound desc = could not find container \"255deca879fdd8ceed95ac7e00f0141370c72414ea7800523990f2cf9ca21611\": container with ID starting with 255deca879fdd8ceed95ac7e00f0141370c72414ea7800523990f2cf9ca21611 not found: ID does not exist"
Jul 29 04:42:44.737270 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:44.737254    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1fcfbd-695769d546-vpdkn"]
Jul 29 04:42:44.746929 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:44.746908    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1fcfbd-78c65659b8-jtr2j"]
Jul 29 04:42:44.750793 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:44.750774    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1fcfbd-78c65659b8-jtr2j"]
Jul 29 04:42:44.760323 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:44.760305    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1fcfbd-c6985bd7b-d49sr"]
Jul 29 04:42:44.769745 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:44.765339    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1fcfbd-c6985bd7b-d49sr"]
Jul 29 04:42:45.835398 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:45.835364    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f3d6632-6e47-447c-81ef-ad97ea1538b8" path="/var/lib/kubelet/pods/3f3d6632-6e47-447c-81ef-ad97ea1538b8/volumes"
Jul 29 04:42:45.835787 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:45.835772    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e6ccb39-07ea-45e1-ad39-5da90124c5ec" path="/var/lib/kubelet/pods/7e6ccb39-07ea-45e1-ad39-5da90124c5ec/volumes"
Jul 29 04:42:45.836113 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:45.836100    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b186e317-2df6-4f9b-a1d8-30fa5c8495e2" path="/var/lib/kubelet/pods/b186e317-2df6-4f9b-a1d8-30fa5c8495e2/volumes"
Jul 29 04:42:48.051959 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:48.051923    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-a32259/test-model-a32259-kserve-6f548cd569-gkvkc" podUID="54c60920-bc3a-4cee-98bb-ec589ae2ac7c" containerName="main" containerID="cri-o://387b5249c4b47e8e40eb6ad12a480ef3f01a65dc9164372598accea71edf82a0" gracePeriod=45
Jul 29 04:42:48.301258 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:48.301235    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a32259/test-model-a32259-kserve-6f548cd569-gkvkc"
Jul 29 04:42:48.454464 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:48.454436    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d9xvq\" (UniqueName: \"kubernetes.io/projected/54c60920-bc3a-4cee-98bb-ec589ae2ac7c-kube-api-access-d9xvq\") pod \"54c60920-bc3a-4cee-98bb-ec589ae2ac7c\" (UID: \"54c60920-bc3a-4cee-98bb-ec589ae2ac7c\") "
Jul 29 04:42:48.454600 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:48.454485    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54c60920-bc3a-4cee-98bb-ec589ae2ac7c-model-cache\") pod \"54c60920-bc3a-4cee-98bb-ec589ae2ac7c\" (UID: \"54c60920-bc3a-4cee-98bb-ec589ae2ac7c\") "
Jul 29 04:42:48.454600 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:48.454524    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54c60920-bc3a-4cee-98bb-ec589ae2ac7c-tmp-dir\") pod \"54c60920-bc3a-4cee-98bb-ec589ae2ac7c\" (UID: \"54c60920-bc3a-4cee-98bb-ec589ae2ac7c\") "
Jul 29 04:42:48.454600 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:48.454572    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54c60920-bc3a-4cee-98bb-ec589ae2ac7c-tls-certs\") pod \"54c60920-bc3a-4cee-98bb-ec589ae2ac7c\" (UID: \"54c60920-bc3a-4cee-98bb-ec589ae2ac7c\") "
Jul 29 04:42:48.454600 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:48.454598    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54c60920-bc3a-4cee-98bb-ec589ae2ac7c-dshm\") pod \"54c60920-bc3a-4cee-98bb-ec589ae2ac7c\" (UID: \"54c60920-bc3a-4cee-98bb-ec589ae2ac7c\") "
Jul 29 04:42:48.454853 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:48.454615    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54c60920-bc3a-4cee-98bb-ec589ae2ac7c-home\") pod \"54c60920-bc3a-4cee-98bb-ec589ae2ac7c\" (UID: \"54c60920-bc3a-4cee-98bb-ec589ae2ac7c\") "
Jul 29 04:42:48.454853 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:48.454771    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54c60920-bc3a-4cee-98bb-ec589ae2ac7c-model-cache" (OuterVolumeSpecName: "model-cache") pod "54c60920-bc3a-4cee-98bb-ec589ae2ac7c" (UID: "54c60920-bc3a-4cee-98bb-ec589ae2ac7c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 04:42:48.454973 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:48.454867    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54c60920-bc3a-4cee-98bb-ec589ae2ac7c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "54c60920-bc3a-4cee-98bb-ec589ae2ac7c" (UID: "54c60920-bc3a-4cee-98bb-ec589ae2ac7c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 04:42:48.454973 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:48.454951    2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54c60920-bc3a-4cee-98bb-ec589ae2ac7c-model-cache\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:42:48.454973 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:48.454958    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54c60920-bc3a-4cee-98bb-ec589ae2ac7c-home" (OuterVolumeSpecName: "home") pod "54c60920-bc3a-4cee-98bb-ec589ae2ac7c" (UID: "54c60920-bc3a-4cee-98bb-ec589ae2ac7c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 04:42:48.456952 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:48.456931    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54c60920-bc3a-4cee-98bb-ec589ae2ac7c-kube-api-access-d9xvq" (OuterVolumeSpecName: "kube-api-access-d9xvq") pod "54c60920-bc3a-4cee-98bb-ec589ae2ac7c" (UID: "54c60920-bc3a-4cee-98bb-ec589ae2ac7c"). InnerVolumeSpecName "kube-api-access-d9xvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 04:42:48.457041 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:48.456948    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54c60920-bc3a-4cee-98bb-ec589ae2ac7c-dshm" (OuterVolumeSpecName: "dshm") pod "54c60920-bc3a-4cee-98bb-ec589ae2ac7c" (UID: "54c60920-bc3a-4cee-98bb-ec589ae2ac7c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 04:42:48.457041 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:48.456950    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54c60920-bc3a-4cee-98bb-ec589ae2ac7c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "54c60920-bc3a-4cee-98bb-ec589ae2ac7c" (UID: "54c60920-bc3a-4cee-98bb-ec589ae2ac7c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 04:42:48.556159 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:48.556128    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9xvq\" (UniqueName: \"kubernetes.io/projected/54c60920-bc3a-4cee-98bb-ec589ae2ac7c-kube-api-access-d9xvq\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:42:48.556159 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:48.556154    2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54c60920-bc3a-4cee-98bb-ec589ae2ac7c-tmp-dir\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:42:48.556159 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:48.556164    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54c60920-bc3a-4cee-98bb-ec589ae2ac7c-tls-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:42:48.556382 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:48.556171    2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54c60920-bc3a-4cee-98bb-ec589ae2ac7c-dshm\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:42:48.556382 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:48.556179    2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54c60920-bc3a-4cee-98bb-ec589ae2ac7c-home\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:42:48.736959 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:48.736874    2570 generic.go:356] "Generic (PLEG): container finished" podID="54c60920-bc3a-4cee-98bb-ec589ae2ac7c" containerID="387b5249c4b47e8e40eb6ad12a480ef3f01a65dc9164372598accea71edf82a0" exitCode=0
Jul 29 04:42:48.736959 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:48.736919    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a32259/test-model-a32259-kserve-6f548cd569-gkvkc"
Jul 29 04:42:48.737120 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:48.736966    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a32259/test-model-a32259-kserve-6f548cd569-gkvkc" event={"ID":"54c60920-bc3a-4cee-98bb-ec589ae2ac7c","Type":"ContainerDied","Data":"387b5249c4b47e8e40eb6ad12a480ef3f01a65dc9164372598accea71edf82a0"}
Jul 29 04:42:48.737120 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:48.737005    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a32259/test-model-a32259-kserve-6f548cd569-gkvkc" event={"ID":"54c60920-bc3a-4cee-98bb-ec589ae2ac7c","Type":"ContainerDied","Data":"7882b4812ae45ec6f0c25a0be74b6b577f4b30335aff16a955fe16384a576c92"}
Jul 29 04:42:48.737120 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:48.737024    2570 scope.go:117] "RemoveContainer" containerID="387b5249c4b47e8e40eb6ad12a480ef3f01a65dc9164372598accea71edf82a0"
Jul 29 04:42:48.746547 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:48.746535    2570 scope.go:117] "RemoveContainer" containerID="387b5249c4b47e8e40eb6ad12a480ef3f01a65dc9164372598accea71edf82a0"
Jul 29 04:42:48.746802 ip-10-0-135-138 kubenswrapper[2570]: E0729 04:42:48.746782    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"387b5249c4b47e8e40eb6ad12a480ef3f01a65dc9164372598accea71edf82a0\": container with ID starting with 387b5249c4b47e8e40eb6ad12a480ef3f01a65dc9164372598accea71edf82a0 not found: ID does not exist" containerID="387b5249c4b47e8e40eb6ad12a480ef3f01a65dc9164372598accea71edf82a0"
Jul 29 04:42:48.746864 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:48.746812    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"387b5249c4b47e8e40eb6ad12a480ef3f01a65dc9164372598accea71edf82a0"} err="failed to get container status \"387b5249c4b47e8e40eb6ad12a480ef3f01a65dc9164372598accea71edf82a0\": rpc error: code = NotFound desc = could not find container \"387b5249c4b47e8e40eb6ad12a480ef3f01a65dc9164372598accea71edf82a0\": container with ID starting with 387b5249c4b47e8e40eb6ad12a480ef3f01a65dc9164372598accea71edf82a0 not found: ID does not exist"
Jul 29 04:42:48.757729 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:48.757701    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-a32259/test-model-a32259-kserve-6f548cd569-gkvkc"]
Jul 29 04:42:48.760605 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:48.760586    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-a32259/test-model-a32259-kserve-6f548cd569-gkvkc"]
Jul 29 04:42:49.835895 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:49.835847    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54c60920-bc3a-4cee-98bb-ec589ae2ac7c" path="/var/lib/kubelet/pods/54c60920-bc3a-4cee-98bb-ec589ae2ac7c/volumes"
Jul 29 04:42:50.685906 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:50.685877    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a32259-kbn4h"]
Jul 29 04:42:50.686269 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:50.686257    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54c60920-bc3a-4cee-98bb-ec589ae2ac7c" containerName="main"
Jul 29 04:42:50.686314 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:50.686271    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="54c60920-bc3a-4cee-98bb-ec589ae2ac7c" containerName="main"
Jul 29 04:42:50.686314 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:50.686289    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f3d6632-6e47-447c-81ef-ad97ea1538b8" containerName="payload-processing"
Jul 29 04:42:50.686314 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:50.686294    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f3d6632-6e47-447c-81ef-ad97ea1538b8" containerName="payload-processing"
Jul 29 04:42:50.686411 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:50.686316    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b186e317-2df6-4f9b-a1d8-30fa5c8495e2" containerName="payload-pre-processing"
Jul 29 04:42:50.686411 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:50.686321    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="b186e317-2df6-4f9b-a1d8-30fa5c8495e2" containerName="payload-pre-processing"
Jul 29 04:42:50.686411 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:50.686328    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e6ccb39-07ea-45e1-ad39-5da90124c5ec" containerName="maas-api"
Jul 29 04:42:50.686411 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:50.686333    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e6ccb39-07ea-45e1-ad39-5da90124c5ec" containerName="maas-api"
Jul 29 04:42:50.686411 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:50.686403    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="b186e317-2df6-4f9b-a1d8-30fa5c8495e2" containerName="payload-pre-processing"
Jul 29 04:42:50.686411 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:50.686411    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e6ccb39-07ea-45e1-ad39-5da90124c5ec" containerName="maas-api"
Jul 29 04:42:50.686591 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:50.686419    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f3d6632-6e47-447c-81ef-ad97ea1538b8" containerName="payload-processing"
Jul 29 04:42:50.686591 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:50.686427    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="54c60920-bc3a-4cee-98bb-ec589ae2ac7c" containerName="main"
Jul 29 04:42:50.690921 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:50.690904    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a32259-kbn4h"
Jul 29 04:42:50.699839 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:50.693776    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 29 04:42:50.699839 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:50.693779    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8frm\""
Jul 29 04:42:50.704329 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:50.703223    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a32259-kbn4h"]
Jul 29 04:42:50.775052 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:50.775018    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2dfe896-675d-4ad8-8145-23e9adb5c715-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a32259-kbn4h\" (UID: \"d2dfe896-675d-4ad8-8145-23e9adb5c715\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a32259-kbn4h"
Jul 29 04:42:50.876387 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:50.876354    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2dfe896-675d-4ad8-8145-23e9adb5c715-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a32259-kbn4h\" (UID: \"d2dfe896-675d-4ad8-8145-23e9adb5c715\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a32259-kbn4h"
Jul 29 04:42:50.877021 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:50.877003    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2dfe896-675d-4ad8-8145-23e9adb5c715-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a32259-kbn4h\" (UID: \"d2dfe896-675d-4ad8-8145-23e9adb5c715\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a32259-kbn4h"
Jul 29 04:42:51.009851 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:51.009753    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a32259-kbn4h"
Jul 29 04:42:51.138528 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:51.138500    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a32259-kbn4h"]
Jul 29 04:42:51.140940 ip-10-0-135-138 kubenswrapper[2570]: W0729 04:42:51.140913    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2dfe896_675d_4ad8_8145_23e9adb5c715.slice/crio-2e8f630f25b4aa1772b3b920b9bd3ebf7917ad3c610b529a9b4db6897d3ac7ac WatchSource:0}: Error finding container 2e8f630f25b4aa1772b3b920b9bd3ebf7917ad3c610b529a9b4db6897d3ac7ac: Status 404 returned error can't find the container with id 2e8f630f25b4aa1772b3b920b9bd3ebf7917ad3c610b529a9b4db6897d3ac7ac
Jul 29 04:42:51.754492 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:51.754455    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a32259-kbn4h" event={"ID":"d2dfe896-675d-4ad8-8145-23e9adb5c715","Type":"ContainerStarted","Data":"d4602720ec15fc51563a9915b569a2c02dcf46af11e671552d3f4b8ecc3766ba"}
Jul 29 04:42:51.754672 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:51.754515    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a32259-kbn4h" event={"ID":"d2dfe896-675d-4ad8-8145-23e9adb5c715","Type":"ContainerStarted","Data":"2e8f630f25b4aa1772b3b920b9bd3ebf7917ad3c610b529a9b4db6897d3ac7ac"}
Jul 29 04:42:51.768773 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:51.768728    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a32259-kbn4h" podStartSLOduration=1.768713371 podStartE2EDuration="1.768713371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:42:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:42:51.768031893 +0000 UTC m=+2836.498162964" watchObservedRunningTime="2026-07-29 04:42:51.768713371 +0000 UTC m=+2836.498844468"
Jul 29 04:42:52.762569 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:52.762529    2570 generic.go:356] "Generic (PLEG): container finished" podID="d2dfe896-675d-4ad8-8145-23e9adb5c715" containerID="d4602720ec15fc51563a9915b569a2c02dcf46af11e671552d3f4b8ecc3766ba" exitCode=0
Jul 29 04:42:52.762986 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:52.762608    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a32259-kbn4h" event={"ID":"d2dfe896-675d-4ad8-8145-23e9adb5c715","Type":"ContainerDied","Data":"d4602720ec15fc51563a9915b569a2c02dcf46af11e671552d3f4b8ecc3766ba"}
Jul 29 04:42:53.898197 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:53.898171    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a32259-kbn4h"
Jul 29 04:42:54.001281 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:54.001254    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2dfe896-675d-4ad8-8145-23e9adb5c715-maas-api-service-ca\") pod \"d2dfe896-675d-4ad8-8145-23e9adb5c715\" (UID: \"d2dfe896-675d-4ad8-8145-23e9adb5c715\") "
Jul 29 04:42:54.001613 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:54.001590    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2dfe896-675d-4ad8-8145-23e9adb5c715-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d2dfe896-675d-4ad8-8145-23e9adb5c715" (UID: "d2dfe896-675d-4ad8-8145-23e9adb5c715"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 04:42:54.102347 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:54.102315    2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2dfe896-675d-4ad8-8145-23e9adb5c715-maas-api-service-ca\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:42:54.773865 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:54.773832    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a32259-kbn4h" event={"ID":"d2dfe896-675d-4ad8-8145-23e9adb5c715","Type":"ContainerDied","Data":"2e8f630f25b4aa1772b3b920b9bd3ebf7917ad3c610b529a9b4db6897d3ac7ac"}
Jul 29 04:42:54.774032 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:54.773875    2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e8f630f25b4aa1772b3b920b9bd3ebf7917ad3c610b529a9b4db6897d3ac7ac"
Jul 29 04:42:54.774032 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:42:54.773875    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a32259-kbn4h"
Jul 29 04:43:01.126913 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:01.126874    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a32259-7b5c7d496d-7dd6f"]
Jul 29 04:43:01.127375 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:01.127109    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-a32259-7b5c7d496d-7dd6f" podUID="7027dd4b-6c64-40c5-b279-3fd8b9937788" containerName="payload-processing" containerID="cri-o://94b616809ed3b4ad41163c265a27d880be6a2e2c40ebc3041e906ddc81e1105c" gracePeriod=30
Jul 29 04:43:01.132386 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:01.132356    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a32259-cdbb84d64-r2qsw"]
Jul 29 04:43:01.135791 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:01.135753    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a32259-cdbb84d64-r2qsw" podUID="fe6ee826-e478-46a5-8e17-02d4e608755d" containerName="payload-pre-processing" containerID="cri-o://a30df8533e53746c31f2ed9a62fff755bdbd4c5e13535e00e72bcfe628d358d8" gracePeriod=30
Jul 29 04:43:01.255868 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:01.255844    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a32259-kbn4h"]
Jul 29 04:43:01.257719 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:01.257690    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a32259-kbn4h"]
Jul 29 04:43:01.432992 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:01.432955    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a32259-7b5c7d496d-7dd6f"
Jul 29 04:43:01.435967 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:01.435944    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a32259-cdbb84d64-r2qsw"
Jul 29 04:43:01.477948 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:01.477338    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7027dd4b-6c64-40c5-b279-3fd8b9937788-plugins-config-volume\") pod \"7027dd4b-6c64-40c5-b279-3fd8b9937788\" (UID: \"7027dd4b-6c64-40c5-b279-3fd8b9937788\") "
Jul 29 04:43:01.477948 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:01.477387    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r4jv7\" (UniqueName: \"kubernetes.io/projected/fe6ee826-e478-46a5-8e17-02d4e608755d-kube-api-access-r4jv7\") pod \"fe6ee826-e478-46a5-8e17-02d4e608755d\" (UID: \"fe6ee826-e478-46a5-8e17-02d4e608755d\") "
Jul 29 04:43:01.477948 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:01.477435    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vrfhc\" (UniqueName: \"kubernetes.io/projected/7027dd4b-6c64-40c5-b279-3fd8b9937788-kube-api-access-vrfhc\") pod \"7027dd4b-6c64-40c5-b279-3fd8b9937788\" (UID: \"7027dd4b-6c64-40c5-b279-3fd8b9937788\") "
Jul 29 04:43:01.477948 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:01.477530    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe6ee826-e478-46a5-8e17-02d4e608755d-plugins-config-volume\") pod \"fe6ee826-e478-46a5-8e17-02d4e608755d\" (UID: \"fe6ee826-e478-46a5-8e17-02d4e608755d\") "
Jul 29 04:43:01.477948 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:01.477749    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7027dd4b-6c64-40c5-b279-3fd8b9937788-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7027dd4b-6c64-40c5-b279-3fd8b9937788" (UID: "7027dd4b-6c64-40c5-b279-3fd8b9937788"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 04:43:01.477948 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:01.477888    2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7027dd4b-6c64-40c5-b279-3fd8b9937788-plugins-config-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:43:01.478395 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:01.478043    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe6ee826-e478-46a5-8e17-02d4e608755d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fe6ee826-e478-46a5-8e17-02d4e608755d" (UID: "fe6ee826-e478-46a5-8e17-02d4e608755d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 04:43:01.480058 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:01.480029    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe6ee826-e478-46a5-8e17-02d4e608755d-kube-api-access-r4jv7" (OuterVolumeSpecName: "kube-api-access-r4jv7") pod "fe6ee826-e478-46a5-8e17-02d4e608755d" (UID: "fe6ee826-e478-46a5-8e17-02d4e608755d"). InnerVolumeSpecName "kube-api-access-r4jv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 04:43:01.480395 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:01.480370    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7027dd4b-6c64-40c5-b279-3fd8b9937788-kube-api-access-vrfhc" (OuterVolumeSpecName: "kube-api-access-vrfhc") pod "7027dd4b-6c64-40c5-b279-3fd8b9937788" (UID: "7027dd4b-6c64-40c5-b279-3fd8b9937788"). InnerVolumeSpecName "kube-api-access-vrfhc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 04:43:01.579353 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:01.579325    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4jv7\" (UniqueName: \"kubernetes.io/projected/fe6ee826-e478-46a5-8e17-02d4e608755d-kube-api-access-r4jv7\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:43:01.579353 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:01.579354    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrfhc\" (UniqueName: \"kubernetes.io/projected/7027dd4b-6c64-40c5-b279-3fd8b9937788-kube-api-access-vrfhc\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:43:01.579544 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:01.579369    2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe6ee826-e478-46a5-8e17-02d4e608755d-plugins-config-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:43:01.811007 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:01.810920    2570 generic.go:356] "Generic (PLEG): container finished" podID="7027dd4b-6c64-40c5-b279-3fd8b9937788" containerID="94b616809ed3b4ad41163c265a27d880be6a2e2c40ebc3041e906ddc81e1105c" exitCode=0
Jul 29 04:43:01.811007 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:01.810972    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a32259-7b5c7d496d-7dd6f"
Jul 29 04:43:01.811209 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:01.811011    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a32259-7b5c7d496d-7dd6f" event={"ID":"7027dd4b-6c64-40c5-b279-3fd8b9937788","Type":"ContainerDied","Data":"94b616809ed3b4ad41163c265a27d880be6a2e2c40ebc3041e906ddc81e1105c"}
Jul 29 04:43:01.811209 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:01.811109    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a32259-7b5c7d496d-7dd6f" event={"ID":"7027dd4b-6c64-40c5-b279-3fd8b9937788","Type":"ContainerDied","Data":"af8419c8a07ee1cecf155b115750c38a5e160494cfb825c0abdd79fde73e15a6"}
Jul 29 04:43:01.811209 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:01.811142    2570 scope.go:117] "RemoveContainer" containerID="94b616809ed3b4ad41163c265a27d880be6a2e2c40ebc3041e906ddc81e1105c"
Jul 29 04:43:01.812434 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:01.812412    2570 generic.go:356] "Generic (PLEG): container finished" podID="fe6ee826-e478-46a5-8e17-02d4e608755d" containerID="a30df8533e53746c31f2ed9a62fff755bdbd4c5e13535e00e72bcfe628d358d8" exitCode=0
Jul 29 04:43:01.812534 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:01.812447    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a32259-cdbb84d64-r2qsw" event={"ID":"fe6ee826-e478-46a5-8e17-02d4e608755d","Type":"ContainerDied","Data":"a30df8533e53746c31f2ed9a62fff755bdbd4c5e13535e00e72bcfe628d358d8"}
Jul 29 04:43:01.812534 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:01.812460    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a32259-cdbb84d64-r2qsw"
Jul 29 04:43:01.812534 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:01.812483    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a32259-cdbb84d64-r2qsw" event={"ID":"fe6ee826-e478-46a5-8e17-02d4e608755d","Type":"ContainerDied","Data":"4a88087adea7f1d710a82dbc84efd1172ebfd810a8a813d5ab969928836a020f"}
Jul 29 04:43:01.820902 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:01.820882    2570 scope.go:117] "RemoveContainer" containerID="94b616809ed3b4ad41163c265a27d880be6a2e2c40ebc3041e906ddc81e1105c"
Jul 29 04:43:01.821187 ip-10-0-135-138 kubenswrapper[2570]: E0729 04:43:01.821170    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94b616809ed3b4ad41163c265a27d880be6a2e2c40ebc3041e906ddc81e1105c\": container with ID starting with 94b616809ed3b4ad41163c265a27d880be6a2e2c40ebc3041e906ddc81e1105c not found: ID does not exist" containerID="94b616809ed3b4ad41163c265a27d880be6a2e2c40ebc3041e906ddc81e1105c"
Jul 29 04:43:01.821262 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:01.821195    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94b616809ed3b4ad41163c265a27d880be6a2e2c40ebc3041e906ddc81e1105c"} err="failed to get container status \"94b616809ed3b4ad41163c265a27d880be6a2e2c40ebc3041e906ddc81e1105c\": rpc error: code = NotFound desc = could not find container \"94b616809ed3b4ad41163c265a27d880be6a2e2c40ebc3041e906ddc81e1105c\": container with ID starting with 94b616809ed3b4ad41163c265a27d880be6a2e2c40ebc3041e906ddc81e1105c not found: ID does not exist"
Jul 29 04:43:01.821262 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:01.821209    2570 scope.go:117] "RemoveContainer" containerID="a30df8533e53746c31f2ed9a62fff755bdbd4c5e13535e00e72bcfe628d358d8"
Jul 29 04:43:01.832729 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:01.832712    2570 scope.go:117] "RemoveContainer" containerID="a30df8533e53746c31f2ed9a62fff755bdbd4c5e13535e00e72bcfe628d358d8"
Jul 29 04:43:01.833030 ip-10-0-135-138 kubenswrapper[2570]: E0729 04:43:01.833013    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a30df8533e53746c31f2ed9a62fff755bdbd4c5e13535e00e72bcfe628d358d8\": container with ID starting with a30df8533e53746c31f2ed9a62fff755bdbd4c5e13535e00e72bcfe628d358d8 not found: ID does not exist" containerID="a30df8533e53746c31f2ed9a62fff755bdbd4c5e13535e00e72bcfe628d358d8"
Jul 29 04:43:01.833107 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:01.833037    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a30df8533e53746c31f2ed9a62fff755bdbd4c5e13535e00e72bcfe628d358d8"} err="failed to get container status \"a30df8533e53746c31f2ed9a62fff755bdbd4c5e13535e00e72bcfe628d358d8\": rpc error: code = NotFound desc = could not find container \"a30df8533e53746c31f2ed9a62fff755bdbd4c5e13535e00e72bcfe628d358d8\": container with ID starting with a30df8533e53746c31f2ed9a62fff755bdbd4c5e13535e00e72bcfe628d358d8 not found: ID does not exist"
Jul 29 04:43:01.837195 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:01.837173    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2dfe896-675d-4ad8-8145-23e9adb5c715" path="/var/lib/kubelet/pods/d2dfe896-675d-4ad8-8145-23e9adb5c715/volumes"
Jul 29 04:43:01.837604 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:01.837571    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a32259-7b5c7d496d-7dd6f"]
Jul 29 04:43:01.839809 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:01.839790    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a32259-7b5c7d496d-7dd6f"]
Jul 29 04:43:01.849629 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:01.849607    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a32259-cdbb84d64-r2qsw"]
Jul 29 04:43:01.852865 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:01.852844    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a32259-cdbb84d64-r2qsw"]
Jul 29 04:43:03.834200 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:03.834165    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7027dd4b-6c64-40c5-b279-3fd8b9937788" path="/var/lib/kubelet/pods/7027dd4b-6c64-40c5-b279-3fd8b9937788/volumes"
Jul 29 04:43:03.834862 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:03.834843    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe6ee826-e478-46a5-8e17-02d4e608755d" path="/var/lib/kubelet/pods/fe6ee826-e478-46a5-8e17-02d4e608755d/volumes"
Jul 29 04:43:11.247697 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:11.247663    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fec70b-588bc697d8-62nvd"]
Jul 29 04:43:11.248189 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:11.248067    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7027dd4b-6c64-40c5-b279-3fd8b9937788" containerName="payload-processing"
Jul 29 04:43:11.248189 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:11.248085    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="7027dd4b-6c64-40c5-b279-3fd8b9937788" containerName="payload-processing"
Jul 29 04:43:11.248189 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:11.248117    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2dfe896-675d-4ad8-8145-23e9adb5c715" containerName="revoke-keys"
Jul 29 04:43:11.248189 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:11.248126    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2dfe896-675d-4ad8-8145-23e9adb5c715" containerName="revoke-keys"
Jul 29 04:43:11.248189 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:11.248156    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe6ee826-e478-46a5-8e17-02d4e608755d" containerName="payload-pre-processing"
Jul 29 04:43:11.248189 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:11.248165    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe6ee826-e478-46a5-8e17-02d4e608755d" containerName="payload-pre-processing"
Jul 29 04:43:11.248520 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:11.248253    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2dfe896-675d-4ad8-8145-23e9adb5c715" containerName="revoke-keys"
Jul 29 04:43:11.248520 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:11.248263    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe6ee826-e478-46a5-8e17-02d4e608755d" containerName="payload-pre-processing"
Jul 29 04:43:11.248520 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:11.248276    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="7027dd4b-6c64-40c5-b279-3fd8b9937788" containerName="payload-processing"
Jul 29 04:43:11.253308 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:11.253288    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fec70b-588bc697d8-62nvd"
Jul 29 04:43:11.255752 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:11.255733    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-fec70b-serving-cert\""
Jul 29 04:43:11.256044 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:11.256027    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-lh6p9\""
Jul 29 04:43:11.256140 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:11.256033    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 29 04:43:11.257094 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:11.257076    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 29 04:43:11.262494 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:11.262464    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fec70b-588bc697d8-62nvd"]
Jul 29 04:43:11.376939 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:11.376901    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hstb4\" (UniqueName: \"kubernetes.io/projected/6c6faed0-01a8-4dea-bc4d-5bcae593f311-kube-api-access-hstb4\") pod \"maas-api-e2e-api-a-fec70b-588bc697d8-62nvd\" (UID: \"6c6faed0-01a8-4dea-bc4d-5bcae593f311\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fec70b-588bc697d8-62nvd"
Jul 29 04:43:11.377099 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:11.376966    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c6faed0-01a8-4dea-bc4d-5bcae593f311-openshift-service-ca\") pod \"maas-api-e2e-api-a-fec70b-588bc697d8-62nvd\" (UID: \"6c6faed0-01a8-4dea-bc4d-5bcae593f311\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fec70b-588bc697d8-62nvd"
Jul 29 04:43:11.377099 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:11.377038    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c6faed0-01a8-4dea-bc4d-5bcae593f311-maas-api-tls\") pod \"maas-api-e2e-api-a-fec70b-588bc697d8-62nvd\" (UID: \"6c6faed0-01a8-4dea-bc4d-5bcae593f311\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fec70b-588bc697d8-62nvd"
Jul 29 04:43:11.478197 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:11.478167    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hstb4\" (UniqueName: \"kubernetes.io/projected/6c6faed0-01a8-4dea-bc4d-5bcae593f311-kube-api-access-hstb4\") pod \"maas-api-e2e-api-a-fec70b-588bc697d8-62nvd\" (UID: \"6c6faed0-01a8-4dea-bc4d-5bcae593f311\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fec70b-588bc697d8-62nvd"
Jul 29 04:43:11.478352 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:11.478222    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c6faed0-01a8-4dea-bc4d-5bcae593f311-openshift-service-ca\") pod \"maas-api-e2e-api-a-fec70b-588bc697d8-62nvd\" (UID: \"6c6faed0-01a8-4dea-bc4d-5bcae593f311\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fec70b-588bc697d8-62nvd"
Jul 29 04:43:11.478352 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:11.478254    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c6faed0-01a8-4dea-bc4d-5bcae593f311-maas-api-tls\") pod \"maas-api-e2e-api-a-fec70b-588bc697d8-62nvd\" (UID: \"6c6faed0-01a8-4dea-bc4d-5bcae593f311\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fec70b-588bc697d8-62nvd"
Jul 29 04:43:11.478427 ip-10-0-135-138 kubenswrapper[2570]: E0729 04:43:11.478378    2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-fec70b-serving-cert: secret "maas-api-e2e-api-a-fec70b-serving-cert" not found
Jul 29 04:43:11.478459 ip-10-0-135-138 kubenswrapper[2570]: E0729 04:43:11.478444    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c6faed0-01a8-4dea-bc4d-5bcae593f311-maas-api-tls podName:6c6faed0-01a8-4dea-bc4d-5bcae593f311 nodeName:}" failed. No retries permitted until 2026-07-29 04:43:11.978427331 +0000 UTC m=+2856.708558381 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6c6faed0-01a8-4dea-bc4d-5bcae593f311-maas-api-tls") pod "maas-api-e2e-api-a-fec70b-588bc697d8-62nvd" (UID: "6c6faed0-01a8-4dea-bc4d-5bcae593f311") : secret "maas-api-e2e-api-a-fec70b-serving-cert" not found
Jul 29 04:43:11.478880 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:11.478860    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c6faed0-01a8-4dea-bc4d-5bcae593f311-openshift-service-ca\") pod \"maas-api-e2e-api-a-fec70b-588bc697d8-62nvd\" (UID: \"6c6faed0-01a8-4dea-bc4d-5bcae593f311\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fec70b-588bc697d8-62nvd"
Jul 29 04:43:11.492468 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:11.492443    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hstb4\" (UniqueName: \"kubernetes.io/projected/6c6faed0-01a8-4dea-bc4d-5bcae593f311-kube-api-access-hstb4\") pod \"maas-api-e2e-api-a-fec70b-588bc697d8-62nvd\" (UID: \"6c6faed0-01a8-4dea-bc4d-5bcae593f311\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fec70b-588bc697d8-62nvd"
Jul 29 04:43:11.846471 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:11.846441    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fec70b-5cb59fb-8bqgd"]
Jul 29 04:43:11.850309 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:11.850290    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fec70b-5cb59fb-8bqgd"
Jul 29 04:43:11.853028 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:11.853002    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-fec70b-dockercfg-bxl9v\""
Jul 29 04:43:11.853141 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:11.853028    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-fec70b\""
Jul 29 04:43:11.863934 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:11.863910    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fec70b-5cb59fb-8bqgd"]
Jul 29 04:43:11.878996 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:11.878975    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fec70b-5bd6f8467d-dpbfq"]
Jul 29 04:43:11.882431 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:11.882416    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-fec70b-5bd6f8467d-dpbfq"
Jul 29 04:43:11.890090 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:11.890065    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fec70b-5bd6f8467d-dpbfq"]
Jul 29 04:43:11.983227 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:11.983190    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpfrx\" (UniqueName: \"kubernetes.io/projected/8f8b902e-b00a-4b34-a796-2e62fabefcaf-kube-api-access-xpfrx\") pod \"payload-pre-processing-e2e-api-a-fec70b-5cb59fb-8bqgd\" (UID: \"8f8b902e-b00a-4b34-a796-2e62fabefcaf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fec70b-5cb59fb-8bqgd"
Jul 29 04:43:11.983379 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:11.983236    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97qpt\" (UniqueName: \"kubernetes.io/projected/4b61fc09-0d50-4853-adb9-caa01675f15f-kube-api-access-97qpt\") pod \"payload-processing-e2e-api-a-fec70b-5bd6f8467d-dpbfq\" (UID: \"4b61fc09-0d50-4853-adb9-caa01675f15f\") " pod="openshift-ingress/payload-processing-e2e-api-a-fec70b-5bd6f8467d-dpbfq"
Jul 29 04:43:11.983379 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:11.983324    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f8b902e-b00a-4b34-a796-2e62fabefcaf-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-fec70b-5cb59fb-8bqgd\" (UID: \"8f8b902e-b00a-4b34-a796-2e62fabefcaf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fec70b-5cb59fb-8bqgd"
Jul 29 04:43:11.983455 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:11.983404    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b61fc09-0d50-4853-adb9-caa01675f15f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-fec70b-5bd6f8467d-dpbfq\" (UID: \"4b61fc09-0d50-4853-adb9-caa01675f15f\") " pod="openshift-ingress/payload-processing-e2e-api-a-fec70b-5bd6f8467d-dpbfq"
Jul 29 04:43:11.983494 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:11.983455    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c6faed0-01a8-4dea-bc4d-5bcae593f311-maas-api-tls\") pod \"maas-api-e2e-api-a-fec70b-588bc697d8-62nvd\" (UID: \"6c6faed0-01a8-4dea-bc4d-5bcae593f311\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fec70b-588bc697d8-62nvd"
Jul 29 04:43:11.985950 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:11.985932    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c6faed0-01a8-4dea-bc4d-5bcae593f311-maas-api-tls\") pod \"maas-api-e2e-api-a-fec70b-588bc697d8-62nvd\" (UID: \"6c6faed0-01a8-4dea-bc4d-5bcae593f311\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fec70b-588bc697d8-62nvd"
Jul 29 04:43:12.084414 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:12.084386    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpfrx\" (UniqueName: \"kubernetes.io/projected/8f8b902e-b00a-4b34-a796-2e62fabefcaf-kube-api-access-xpfrx\") pod \"payload-pre-processing-e2e-api-a-fec70b-5cb59fb-8bqgd\" (UID: \"8f8b902e-b00a-4b34-a796-2e62fabefcaf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fec70b-5cb59fb-8bqgd"
Jul 29 04:43:12.084546 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:12.084426    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97qpt\" (UniqueName: \"kubernetes.io/projected/4b61fc09-0d50-4853-adb9-caa01675f15f-kube-api-access-97qpt\") pod \"payload-processing-e2e-api-a-fec70b-5bd6f8467d-dpbfq\" (UID: \"4b61fc09-0d50-4853-adb9-caa01675f15f\") " pod="openshift-ingress/payload-processing-e2e-api-a-fec70b-5bd6f8467d-dpbfq"
Jul 29 04:43:12.084546 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:12.084468    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f8b902e-b00a-4b34-a796-2e62fabefcaf-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-fec70b-5cb59fb-8bqgd\" (UID: \"8f8b902e-b00a-4b34-a796-2e62fabefcaf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fec70b-5cb59fb-8bqgd"
Jul 29 04:43:12.084546 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:12.084523    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b61fc09-0d50-4853-adb9-caa01675f15f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-fec70b-5bd6f8467d-dpbfq\" (UID: \"4b61fc09-0d50-4853-adb9-caa01675f15f\") " pod="openshift-ingress/payload-processing-e2e-api-a-fec70b-5bd6f8467d-dpbfq"
Jul 29 04:43:12.085401 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:12.085382    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f8b902e-b00a-4b34-a796-2e62fabefcaf-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-fec70b-5cb59fb-8bqgd\" (UID: \"8f8b902e-b00a-4b34-a796-2e62fabefcaf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fec70b-5cb59fb-8bqgd"
Jul 29 04:43:12.085401 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:12.085391    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b61fc09-0d50-4853-adb9-caa01675f15f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-fec70b-5bd6f8467d-dpbfq\" (UID: \"4b61fc09-0d50-4853-adb9-caa01675f15f\") " pod="openshift-ingress/payload-processing-e2e-api-a-fec70b-5bd6f8467d-dpbfq"
Jul 29 04:43:12.092361 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:12.092338    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97qpt\" (UniqueName: \"kubernetes.io/projected/4b61fc09-0d50-4853-adb9-caa01675f15f-kube-api-access-97qpt\") pod \"payload-processing-e2e-api-a-fec70b-5bd6f8467d-dpbfq\" (UID: \"4b61fc09-0d50-4853-adb9-caa01675f15f\") " pod="openshift-ingress/payload-processing-e2e-api-a-fec70b-5bd6f8467d-dpbfq"
Jul 29 04:43:12.092458 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:12.092386    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpfrx\" (UniqueName: \"kubernetes.io/projected/8f8b902e-b00a-4b34-a796-2e62fabefcaf-kube-api-access-xpfrx\") pod \"payload-pre-processing-e2e-api-a-fec70b-5cb59fb-8bqgd\" (UID: \"8f8b902e-b00a-4b34-a796-2e62fabefcaf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fec70b-5cb59fb-8bqgd"
Jul 29 04:43:12.161165 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:12.161098    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fec70b-5cb59fb-8bqgd"
Jul 29 04:43:12.166141 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:12.166122    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fec70b-588bc697d8-62nvd"
Jul 29 04:43:12.193077 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:12.193041    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-fec70b-5bd6f8467d-dpbfq"
Jul 29 04:43:12.323652 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:12.323620    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fec70b-588bc697d8-62nvd"]
Jul 29 04:43:12.325772 ip-10-0-135-138 kubenswrapper[2570]: W0729 04:43:12.325742    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c6faed0_01a8_4dea_bc4d_5bcae593f311.slice/crio-e16a1ebba9961d45a0929b5a39f88c7d6b12955493e8f62d7a3f702009b23280 WatchSource:0}: Error finding container e16a1ebba9961d45a0929b5a39f88c7d6b12955493e8f62d7a3f702009b23280: Status 404 returned error can't find the container with id e16a1ebba9961d45a0929b5a39f88c7d6b12955493e8f62d7a3f702009b23280
Jul 29 04:43:12.372735 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:12.372710    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fec70b-5bd6f8467d-dpbfq"]
Jul 29 04:43:12.374143 ip-10-0-135-138 kubenswrapper[2570]: W0729 04:43:12.374116    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b61fc09_0d50_4853_adb9_caa01675f15f.slice/crio-52988292bfc188726a945109635a99f6ed48b6c87da0b2dd42b70c975be8ab5d WatchSource:0}: Error finding container 52988292bfc188726a945109635a99f6ed48b6c87da0b2dd42b70c975be8ab5d: Status 404 returned error can't find the container with id 52988292bfc188726a945109635a99f6ed48b6c87da0b2dd42b70c975be8ab5d
Jul 29 04:43:12.549909 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:12.549884    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fec70b-5cb59fb-8bqgd"]
Jul 29 04:43:12.551689 ip-10-0-135-138 kubenswrapper[2570]: W0729 04:43:12.551657    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f8b902e_b00a_4b34_a796_2e62fabefcaf.slice/crio-bad10904da8c2be55dcefb950963ce465c3c1de887fe6eceae166d19317d968a WatchSource:0}: Error finding container bad10904da8c2be55dcefb950963ce465c3c1de887fe6eceae166d19317d968a: Status 404 returned error can't find the container with id bad10904da8c2be55dcefb950963ce465c3c1de887fe6eceae166d19317d968a
Jul 29 04:43:12.870973 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:12.870939    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fec70b-588bc697d8-62nvd" event={"ID":"6c6faed0-01a8-4dea-bc4d-5bcae593f311","Type":"ContainerStarted","Data":"6299f85cc35e94b8bf2e939f3ec0c5ba9076dcb9a1359688fde854f80433f4fc"}
Jul 29 04:43:12.871142 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:12.870995    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fec70b-588bc697d8-62nvd" event={"ID":"6c6faed0-01a8-4dea-bc4d-5bcae593f311","Type":"ContainerStarted","Data":"e16a1ebba9961d45a0929b5a39f88c7d6b12955493e8f62d7a3f702009b23280"}
Jul 29 04:43:12.871142 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:12.871044    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fec70b-588bc697d8-62nvd"
Jul 29 04:43:12.872609 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:12.872587    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-fec70b-5bd6f8467d-dpbfq" event={"ID":"4b61fc09-0d50-4853-adb9-caa01675f15f","Type":"ContainerStarted","Data":"4b838d16503cf17a7c69852fcc2da23222a4c4a9d453e51ea39577a29805877e"}
Jul 29 04:43:12.872713 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:12.872626    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-fec70b-5bd6f8467d-dpbfq" event={"ID":"4b61fc09-0d50-4853-adb9-caa01675f15f","Type":"ContainerStarted","Data":"52988292bfc188726a945109635a99f6ed48b6c87da0b2dd42b70c975be8ab5d"}
Jul 29 04:43:12.872713 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:12.872670    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-fec70b-5bd6f8467d-dpbfq"
Jul 29 04:43:12.874046 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:12.874015    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fec70b-5cb59fb-8bqgd" event={"ID":"8f8b902e-b00a-4b34-a796-2e62fabefcaf","Type":"ContainerStarted","Data":"62f6b6c89f1638937c3bc7bc838dbca2354943700273799eb152719fc8800f47"}
Jul 29 04:43:12.874126 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:12.874056    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fec70b-5cb59fb-8bqgd" event={"ID":"8f8b902e-b00a-4b34-a796-2e62fabefcaf","Type":"ContainerStarted","Data":"bad10904da8c2be55dcefb950963ce465c3c1de887fe6eceae166d19317d968a"}
Jul 29 04:43:12.874126 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:12.874092    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fec70b-5cb59fb-8bqgd"
Jul 29 04:43:12.887597 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:12.887553    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fec70b-588bc697d8-62nvd" podStartSLOduration=1.457610249 podStartE2EDuration="1.887541438s" totalImagesPullingTime="429.931189ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:43:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:43:12.886373136 +0000 UTC m=+2857.616504220" watchObservedRunningTime="2026-07-29 04:43:12.887541438 +0000 UTC m=+2857.617672508"
Jul 29 04:43:12.900036 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:12.899995    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fec70b-5cb59fb-8bqgd" podStartSLOduration=1.899981521 podStartE2EDuration="1.899981521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:43:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:43:12.899963354 +0000 UTC m=+2857.630094442" watchObservedRunningTime="2026-07-29 04:43:12.899981521 +0000 UTC m=+2857.630112592"
Jul 29 04:43:12.917006 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:12.916960    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-fec70b-5bd6f8467d-dpbfq" podStartSLOduration=1.916943745 podStartE2EDuration="1.916943745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:43:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:43:12.914473706 +0000 UTC m=+2857.644604811" watchObservedRunningTime="2026-07-29 04:43:12.916943745 +0000 UTC m=+2857.647074815"
Jul 29 04:43:18.884392 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:18.884360    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fec70b-588bc697d8-62nvd"
Jul 29 04:43:23.881351 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:23.881321    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fec70b-5cb59fb-8bqgd"
Jul 29 04:43:23.883449 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:23.881385    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-fec70b-5bd6f8467d-dpbfq"
Jul 29 04:43:36.096171 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:36.096099    2570 scope.go:117] "RemoveContainer" containerID="e80fddcd2acc4e4657d953aca7b28a93c2024c7a1112d84739a63c5574c66463"
Jul 29 04:43:39.414952 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:39.414919    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a099ac-64c95c68d9-hghgt"]
Jul 29 04:43:39.420432 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:39.420412    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a099ac-64c95c68d9-hghgt"
Jul 29 04:43:39.422985 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:39.422955    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-a099ac-serving-cert\""
Jul 29 04:43:39.429681 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:39.429655    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a099ac-64c95c68d9-hghgt"]
Jul 29 04:43:39.530873 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:39.530843    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abe81f5f-dae4-4b5c-8a5b-9cccc5296213-maas-api-tls\") pod \"maas-api-e2e-api-b-a099ac-64c95c68d9-hghgt\" (UID: \"abe81f5f-dae4-4b5c-8a5b-9cccc5296213\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a099ac-64c95c68d9-hghgt"
Jul 29 04:43:39.531025 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:39.530913    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abe81f5f-dae4-4b5c-8a5b-9cccc5296213-openshift-service-ca\") pod \"maas-api-e2e-api-b-a099ac-64c95c68d9-hghgt\" (UID: \"abe81f5f-dae4-4b5c-8a5b-9cccc5296213\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a099ac-64c95c68d9-hghgt"
Jul 29 04:43:39.531025 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:39.530948    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frtgv\" (UniqueName: \"kubernetes.io/projected/abe81f5f-dae4-4b5c-8a5b-9cccc5296213-kube-api-access-frtgv\") pod \"maas-api-e2e-api-b-a099ac-64c95c68d9-hghgt\" (UID: \"abe81f5f-dae4-4b5c-8a5b-9cccc5296213\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a099ac-64c95c68d9-hghgt"
Jul 29 04:43:39.632117 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:39.632088    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abe81f5f-dae4-4b5c-8a5b-9cccc5296213-maas-api-tls\") pod \"maas-api-e2e-api-b-a099ac-64c95c68d9-hghgt\" (UID: \"abe81f5f-dae4-4b5c-8a5b-9cccc5296213\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a099ac-64c95c68d9-hghgt"
Jul 29 04:43:39.632264 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:39.632152    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abe81f5f-dae4-4b5c-8a5b-9cccc5296213-openshift-service-ca\") pod \"maas-api-e2e-api-b-a099ac-64c95c68d9-hghgt\" (UID: \"abe81f5f-dae4-4b5c-8a5b-9cccc5296213\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a099ac-64c95c68d9-hghgt"
Jul 29 04:43:39.632264 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:39.632186    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frtgv\" (UniqueName: \"kubernetes.io/projected/abe81f5f-dae4-4b5c-8a5b-9cccc5296213-kube-api-access-frtgv\") pod \"maas-api-e2e-api-b-a099ac-64c95c68d9-hghgt\" (UID: \"abe81f5f-dae4-4b5c-8a5b-9cccc5296213\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a099ac-64c95c68d9-hghgt"
Jul 29 04:43:39.632264 ip-10-0-135-138 kubenswrapper[2570]: E0729 04:43:39.632251    2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-a099ac-serving-cert: secret "maas-api-e2e-api-b-a099ac-serving-cert" not found
Jul 29 04:43:39.632365 ip-10-0-135-138 kubenswrapper[2570]: E0729 04:43:39.632326    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abe81f5f-dae4-4b5c-8a5b-9cccc5296213-maas-api-tls podName:abe81f5f-dae4-4b5c-8a5b-9cccc5296213 nodeName:}" failed. No retries permitted until 2026-07-29 04:43:40.132308049 +0000 UTC m=+2884.862439100 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/abe81f5f-dae4-4b5c-8a5b-9cccc5296213-maas-api-tls") pod "maas-api-e2e-api-b-a099ac-64c95c68d9-hghgt" (UID: "abe81f5f-dae4-4b5c-8a5b-9cccc5296213") : secret "maas-api-e2e-api-b-a099ac-serving-cert" not found
Jul 29 04:43:39.632851 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:39.632807    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abe81f5f-dae4-4b5c-8a5b-9cccc5296213-openshift-service-ca\") pod \"maas-api-e2e-api-b-a099ac-64c95c68d9-hghgt\" (UID: \"abe81f5f-dae4-4b5c-8a5b-9cccc5296213\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a099ac-64c95c68d9-hghgt"
Jul 29 04:43:39.640416 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:39.640391    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frtgv\" (UniqueName: \"kubernetes.io/projected/abe81f5f-dae4-4b5c-8a5b-9cccc5296213-kube-api-access-frtgv\") pod \"maas-api-e2e-api-b-a099ac-64c95c68d9-hghgt\" (UID: \"abe81f5f-dae4-4b5c-8a5b-9cccc5296213\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a099ac-64c95c68d9-hghgt"
Jul 29 04:43:40.135943 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:40.135913    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abe81f5f-dae4-4b5c-8a5b-9cccc5296213-maas-api-tls\") pod \"maas-api-e2e-api-b-a099ac-64c95c68d9-hghgt\" (UID: \"abe81f5f-dae4-4b5c-8a5b-9cccc5296213\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a099ac-64c95c68d9-hghgt"
Jul 29 04:43:40.138447 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:40.138422    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abe81f5f-dae4-4b5c-8a5b-9cccc5296213-maas-api-tls\") pod \"maas-api-e2e-api-b-a099ac-64c95c68d9-hghgt\" (UID: \"abe81f5f-dae4-4b5c-8a5b-9cccc5296213\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a099ac-64c95c68d9-hghgt"
Jul 29 04:43:40.201446 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:40.201418    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a099ac-658c674bcb-72skk"]
Jul 29 04:43:40.205376 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:40.205357    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a099ac-658c674bcb-72skk"
Jul 29 04:43:40.207873 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:40.207853    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-a099ac\""
Jul 29 04:43:40.215762 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:40.215742    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a099ac-658c674bcb-72skk"]
Jul 29 04:43:40.236835 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:40.236795    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hjxm\" (UniqueName: \"kubernetes.io/projected/595da500-eb7b-4a3f-aac7-00d328153825-kube-api-access-7hjxm\") pod \"payload-pre-processing-e2e-api-b-a099ac-658c674bcb-72skk\" (UID: \"595da500-eb7b-4a3f-aac7-00d328153825\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a099ac-658c674bcb-72skk"
Jul 29 04:43:40.236958 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:40.236941    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/595da500-eb7b-4a3f-aac7-00d328153825-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a099ac-658c674bcb-72skk\" (UID: \"595da500-eb7b-4a3f-aac7-00d328153825\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a099ac-658c674bcb-72skk"
Jul 29 04:43:40.332565 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:40.332537    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a099ac-64c95c68d9-hghgt"
Jul 29 04:43:40.337747 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:40.337721    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/595da500-eb7b-4a3f-aac7-00d328153825-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a099ac-658c674bcb-72skk\" (UID: \"595da500-eb7b-4a3f-aac7-00d328153825\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a099ac-658c674bcb-72skk"
Jul 29 04:43:40.337920 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:40.337779    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hjxm\" (UniqueName: \"kubernetes.io/projected/595da500-eb7b-4a3f-aac7-00d328153825-kube-api-access-7hjxm\") pod \"payload-pre-processing-e2e-api-b-a099ac-658c674bcb-72skk\" (UID: \"595da500-eb7b-4a3f-aac7-00d328153825\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a099ac-658c674bcb-72skk"
Jul 29 04:43:40.338426 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:40.338403    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/595da500-eb7b-4a3f-aac7-00d328153825-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a099ac-658c674bcb-72skk\" (UID: \"595da500-eb7b-4a3f-aac7-00d328153825\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a099ac-658c674bcb-72skk"
Jul 29 04:43:40.346672 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:40.346652    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hjxm\" (UniqueName: \"kubernetes.io/projected/595da500-eb7b-4a3f-aac7-00d328153825-kube-api-access-7hjxm\") pod \"payload-pre-processing-e2e-api-b-a099ac-658c674bcb-72skk\" (UID: \"595da500-eb7b-4a3f-aac7-00d328153825\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a099ac-658c674bcb-72skk"
Jul 29 04:43:40.470858 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:40.470802    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a099ac-64c95c68d9-hghgt"]
Jul 29 04:43:40.473362 ip-10-0-135-138 kubenswrapper[2570]: W0729 04:43:40.473333    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabe81f5f_dae4_4b5c_8a5b_9cccc5296213.slice/crio-ae425993586ffb3e19ab2f10c7f26318faf1513d205ec4dbb77f0cec3225b326 WatchSource:0}: Error finding container ae425993586ffb3e19ab2f10c7f26318faf1513d205ec4dbb77f0cec3225b326: Status 404 returned error can't find the container with id ae425993586ffb3e19ab2f10c7f26318faf1513d205ec4dbb77f0cec3225b326
Jul 29 04:43:40.519721 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:40.519697    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a099ac-658c674bcb-72skk"
Jul 29 04:43:40.656428 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:40.656404    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a099ac-658c674bcb-72skk"]
Jul 29 04:43:40.658410 ip-10-0-135-138 kubenswrapper[2570]: W0729 04:43:40.658389    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod595da500_eb7b_4a3f_aac7_00d328153825.slice/crio-649c6026ebc1553dafd6c6698f5dda5099c44dff3b91d696fd8711e510e75ef0 WatchSource:0}: Error finding container 649c6026ebc1553dafd6c6698f5dda5099c44dff3b91d696fd8711e510e75ef0: Status 404 returned error can't find the container with id 649c6026ebc1553dafd6c6698f5dda5099c44dff3b91d696fd8711e510e75ef0
Jul 29 04:43:41.043852 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:41.043740    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a099ac-64c95c68d9-hghgt" event={"ID":"abe81f5f-dae4-4b5c-8a5b-9cccc5296213","Type":"ContainerStarted","Data":"035650d65d751054453efb4470be9bf30d420d9fdc422be0ca432142068f4f27"}
Jul 29 04:43:41.044017 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:41.043851    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a099ac-64c95c68d9-hghgt" event={"ID":"abe81f5f-dae4-4b5c-8a5b-9cccc5296213","Type":"ContainerStarted","Data":"ae425993586ffb3e19ab2f10c7f26318faf1513d205ec4dbb77f0cec3225b326"}
Jul 29 04:43:41.044017 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:41.043952    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a099ac-64c95c68d9-hghgt"
Jul 29 04:43:41.045358 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:41.045333    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a099ac-658c674bcb-72skk" event={"ID":"595da500-eb7b-4a3f-aac7-00d328153825","Type":"ContainerStarted","Data":"ca11f4f8f89ffaba0c35c46a03704276b87df02cb5dea5947baa2140194eed1f"}
Jul 29 04:43:41.045464 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:41.045373    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a099ac-658c674bcb-72skk" event={"ID":"595da500-eb7b-4a3f-aac7-00d328153825","Type":"ContainerStarted","Data":"649c6026ebc1553dafd6c6698f5dda5099c44dff3b91d696fd8711e510e75ef0"}
Jul 29 04:43:41.045509 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:41.045483    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a099ac-658c674bcb-72skk"
Jul 29 04:43:41.061131 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:41.061085    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a099ac-64c95c68d9-hghgt" podStartSLOduration=1.749957279 podStartE2EDuration="2.061071385s" totalImagesPullingTime="311.114106ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:43:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:43:41.059262941 +0000 UTC m=+2885.789394029" watchObservedRunningTime="2026-07-29 04:43:41.061071385 +0000 UTC m=+2885.791202454"
Jul 29 04:43:41.075486 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:41.075446    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a099ac-658c674bcb-72skk" podStartSLOduration=1.075433327 podStartE2EDuration="1.075433327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:43:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:43:41.073396609 +0000 UTC m=+2885.803527704" watchObservedRunningTime="2026-07-29 04:43:41.075433327 +0000 UTC m=+2885.805564397"
Jul 29 04:43:47.055724 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:47.055694    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a099ac-64c95c68d9-hghgt"
Jul 29 04:43:52.052919 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:43:52.052884    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a099ac-658c674bcb-72skk"
Jul 29 04:44:22.626199 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:22.626167    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fec70b-588bc697d8-62nvd"]
Jul 29 04:44:22.626844 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:22.626522    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fec70b-588bc697d8-62nvd" podUID="6c6faed0-01a8-4dea-bc4d-5bcae593f311" containerName="maas-api" containerID="cri-o://6299f85cc35e94b8bf2e939f3ec0c5ba9076dcb9a1359688fde854f80433f4fc" gracePeriod=30
Jul 29 04:44:22.675285 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:22.675253    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fec70b-5bd6f8467d-dpbfq"]
Jul 29 04:44:22.675516 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:22.675488    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-fec70b-5bd6f8467d-dpbfq" podUID="4b61fc09-0d50-4853-adb9-caa01675f15f" containerName="payload-processing" containerID="cri-o://4b838d16503cf17a7c69852fcc2da23222a4c4a9d453e51ea39577a29805877e" gracePeriod=30
Jul 29 04:44:22.683471 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:22.683443    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fec70b-5cb59fb-8bqgd"]
Jul 29 04:44:22.683715 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:22.683693    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fec70b-5cb59fb-8bqgd" podUID="8f8b902e-b00a-4b34-a796-2e62fabefcaf" containerName="payload-pre-processing" containerID="cri-o://62f6b6c89f1638937c3bc7bc838dbca2354943700273799eb152719fc8800f47" gracePeriod=30
Jul 29 04:44:22.964422 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:22.964387    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-fec70b-5bd6f8467d-dpbfq"
Jul 29 04:44:22.967508 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:22.967485    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fec70b-588bc697d8-62nvd"
Jul 29 04:44:22.987204 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:22.987184    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fec70b-5cb59fb-8bqgd"
Jul 29 04:44:23.013519 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:23.013493    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xpfrx\" (UniqueName: \"kubernetes.io/projected/8f8b902e-b00a-4b34-a796-2e62fabefcaf-kube-api-access-xpfrx\") pod \"8f8b902e-b00a-4b34-a796-2e62fabefcaf\" (UID: \"8f8b902e-b00a-4b34-a796-2e62fabefcaf\") "
Jul 29 04:44:23.013654 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:23.013533    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c6faed0-01a8-4dea-bc4d-5bcae593f311-maas-api-tls\") pod \"6c6faed0-01a8-4dea-bc4d-5bcae593f311\" (UID: \"6c6faed0-01a8-4dea-bc4d-5bcae593f311\") "
Jul 29 04:44:23.013654 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:23.013555    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-97qpt\" (UniqueName: \"kubernetes.io/projected/4b61fc09-0d50-4853-adb9-caa01675f15f-kube-api-access-97qpt\") pod \"4b61fc09-0d50-4853-adb9-caa01675f15f\" (UID: \"4b61fc09-0d50-4853-adb9-caa01675f15f\") "
Jul 29 04:44:23.013654 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:23.013609    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c6faed0-01a8-4dea-bc4d-5bcae593f311-openshift-service-ca\") pod \"6c6faed0-01a8-4dea-bc4d-5bcae593f311\" (UID: \"6c6faed0-01a8-4dea-bc4d-5bcae593f311\") "
Jul 29 04:44:23.013654 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:23.013637    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hstb4\" (UniqueName: \"kubernetes.io/projected/6c6faed0-01a8-4dea-bc4d-5bcae593f311-kube-api-access-hstb4\") pod \"6c6faed0-01a8-4dea-bc4d-5bcae593f311\" (UID: \"6c6faed0-01a8-4dea-bc4d-5bcae593f311\") "
Jul 29 04:44:23.013970 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:23.013688    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f8b902e-b00a-4b34-a796-2e62fabefcaf-plugins-config-volume\") pod \"8f8b902e-b00a-4b34-a796-2e62fabefcaf\" (UID: \"8f8b902e-b00a-4b34-a796-2e62fabefcaf\") "
Jul 29 04:44:23.013970 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:23.013762    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b61fc09-0d50-4853-adb9-caa01675f15f-plugins-config-volume\") pod \"4b61fc09-0d50-4853-adb9-caa01675f15f\" (UID: \"4b61fc09-0d50-4853-adb9-caa01675f15f\") "
Jul 29 04:44:23.014238 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:23.014189    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f8b902e-b00a-4b34-a796-2e62fabefcaf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8f8b902e-b00a-4b34-a796-2e62fabefcaf" (UID: "8f8b902e-b00a-4b34-a796-2e62fabefcaf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 04:44:23.015801 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:23.014650    2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f8b902e-b00a-4b34-a796-2e62fabefcaf-plugins-config-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:44:23.015801 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:23.015261    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c6faed0-01a8-4dea-bc4d-5bcae593f311-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6c6faed0-01a8-4dea-bc4d-5bcae593f311" (UID: "6c6faed0-01a8-4dea-bc4d-5bcae593f311"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 04:44:23.015801 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:23.015742    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b61fc09-0d50-4853-adb9-caa01675f15f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4b61fc09-0d50-4853-adb9-caa01675f15f" (UID: "4b61fc09-0d50-4853-adb9-caa01675f15f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 04:44:23.016397 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:23.016377    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c6faed0-01a8-4dea-bc4d-5bcae593f311-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6c6faed0-01a8-4dea-bc4d-5bcae593f311" (UID: "6c6faed0-01a8-4dea-bc4d-5bcae593f311"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 04:44:23.016508 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:23.016489    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f8b902e-b00a-4b34-a796-2e62fabefcaf-kube-api-access-xpfrx" (OuterVolumeSpecName: "kube-api-access-xpfrx") pod "8f8b902e-b00a-4b34-a796-2e62fabefcaf" (UID: "8f8b902e-b00a-4b34-a796-2e62fabefcaf"). InnerVolumeSpecName "kube-api-access-xpfrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 04:44:23.016660 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:23.016637    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c6faed0-01a8-4dea-bc4d-5bcae593f311-kube-api-access-hstb4" (OuterVolumeSpecName: "kube-api-access-hstb4") pod "6c6faed0-01a8-4dea-bc4d-5bcae593f311" (UID: "6c6faed0-01a8-4dea-bc4d-5bcae593f311"). InnerVolumeSpecName "kube-api-access-hstb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 04:44:23.016934 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:23.016908    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b61fc09-0d50-4853-adb9-caa01675f15f-kube-api-access-97qpt" (OuterVolumeSpecName: "kube-api-access-97qpt") pod "4b61fc09-0d50-4853-adb9-caa01675f15f" (UID: "4b61fc09-0d50-4853-adb9-caa01675f15f"). InnerVolumeSpecName "kube-api-access-97qpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 04:44:23.116057 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:23.116023    2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c6faed0-01a8-4dea-bc4d-5bcae593f311-openshift-service-ca\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:44:23.116057 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:23.116051    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hstb4\" (UniqueName: \"kubernetes.io/projected/6c6faed0-01a8-4dea-bc4d-5bcae593f311-kube-api-access-hstb4\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:44:23.116244 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:23.116066    2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b61fc09-0d50-4853-adb9-caa01675f15f-plugins-config-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:44:23.116244 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:23.116080    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpfrx\" (UniqueName: \"kubernetes.io/projected/8f8b902e-b00a-4b34-a796-2e62fabefcaf-kube-api-access-xpfrx\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:44:23.116244 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:23.116092    2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c6faed0-01a8-4dea-bc4d-5bcae593f311-maas-api-tls\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:44:23.116244 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:23.116102    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97qpt\" (UniqueName: \"kubernetes.io/projected/4b61fc09-0d50-4853-adb9-caa01675f15f-kube-api-access-97qpt\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:44:23.266790 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:23.266706    2570 generic.go:356] "Generic (PLEG): container finished" podID="6c6faed0-01a8-4dea-bc4d-5bcae593f311" containerID="6299f85cc35e94b8bf2e939f3ec0c5ba9076dcb9a1359688fde854f80433f4fc" exitCode=0
Jul 29 04:44:23.266790 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:23.266755    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fec70b-588bc697d8-62nvd"
Jul 29 04:44:23.267006 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:23.266796    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fec70b-588bc697d8-62nvd" event={"ID":"6c6faed0-01a8-4dea-bc4d-5bcae593f311","Type":"ContainerDied","Data":"6299f85cc35e94b8bf2e939f3ec0c5ba9076dcb9a1359688fde854f80433f4fc"}
Jul 29 04:44:23.267006 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:23.266871    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fec70b-588bc697d8-62nvd" event={"ID":"6c6faed0-01a8-4dea-bc4d-5bcae593f311","Type":"ContainerDied","Data":"e16a1ebba9961d45a0929b5a39f88c7d6b12955493e8f62d7a3f702009b23280"}
Jul 29 04:44:23.267006 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:23.266905    2570 scope.go:117] "RemoveContainer" containerID="6299f85cc35e94b8bf2e939f3ec0c5ba9076dcb9a1359688fde854f80433f4fc"
Jul 29 04:44:23.268237 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:23.268212    2570 generic.go:356] "Generic (PLEG): container finished" podID="4b61fc09-0d50-4853-adb9-caa01675f15f" containerID="4b838d16503cf17a7c69852fcc2da23222a4c4a9d453e51ea39577a29805877e" exitCode=0
Jul 29 04:44:23.268340 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:23.268250    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-fec70b-5bd6f8467d-dpbfq"
Jul 29 04:44:23.268340 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:23.268296    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-fec70b-5bd6f8467d-dpbfq" event={"ID":"4b61fc09-0d50-4853-adb9-caa01675f15f","Type":"ContainerDied","Data":"4b838d16503cf17a7c69852fcc2da23222a4c4a9d453e51ea39577a29805877e"}
Jul 29 04:44:23.268444 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:23.268345    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-fec70b-5bd6f8467d-dpbfq" event={"ID":"4b61fc09-0d50-4853-adb9-caa01675f15f","Type":"ContainerDied","Data":"52988292bfc188726a945109635a99f6ed48b6c87da0b2dd42b70c975be8ab5d"}
Jul 29 04:44:23.269922 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:23.269898    2570 generic.go:356] "Generic (PLEG): container finished" podID="8f8b902e-b00a-4b34-a796-2e62fabefcaf" containerID="62f6b6c89f1638937c3bc7bc838dbca2354943700273799eb152719fc8800f47" exitCode=0
Jul 29 04:44:23.270018 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:23.269934    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fec70b-5cb59fb-8bqgd" event={"ID":"8f8b902e-b00a-4b34-a796-2e62fabefcaf","Type":"ContainerDied","Data":"62f6b6c89f1638937c3bc7bc838dbca2354943700273799eb152719fc8800f47"}
Jul 29 04:44:23.270018 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:23.269983    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fec70b-5cb59fb-8bqgd" event={"ID":"8f8b902e-b00a-4b34-a796-2e62fabefcaf","Type":"ContainerDied","Data":"bad10904da8c2be55dcefb950963ce465c3c1de887fe6eceae166d19317d968a"}
Jul 29 04:44:23.270018 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:23.269944    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fec70b-5cb59fb-8bqgd"
Jul 29 04:44:23.277310 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:23.277290    2570 scope.go:117] "RemoveContainer" containerID="6299f85cc35e94b8bf2e939f3ec0c5ba9076dcb9a1359688fde854f80433f4fc"
Jul 29 04:44:23.277679 ip-10-0-135-138 kubenswrapper[2570]: E0729 04:44:23.277655    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6299f85cc35e94b8bf2e939f3ec0c5ba9076dcb9a1359688fde854f80433f4fc\": container with ID starting with 6299f85cc35e94b8bf2e939f3ec0c5ba9076dcb9a1359688fde854f80433f4fc not found: ID does not exist" containerID="6299f85cc35e94b8bf2e939f3ec0c5ba9076dcb9a1359688fde854f80433f4fc"
Jul 29 04:44:23.277759 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:23.277686    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6299f85cc35e94b8bf2e939f3ec0c5ba9076dcb9a1359688fde854f80433f4fc"} err="failed to get container status \"6299f85cc35e94b8bf2e939f3ec0c5ba9076dcb9a1359688fde854f80433f4fc\": rpc error: code = NotFound desc = could not find container \"6299f85cc35e94b8bf2e939f3ec0c5ba9076dcb9a1359688fde854f80433f4fc\": container with ID starting with 6299f85cc35e94b8bf2e939f3ec0c5ba9076dcb9a1359688fde854f80433f4fc not found: ID does not exist"
Jul 29 04:44:23.277759 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:23.277727    2570 scope.go:117] "RemoveContainer" containerID="4b838d16503cf17a7c69852fcc2da23222a4c4a9d453e51ea39577a29805877e"
Jul 29 04:44:23.286434 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:23.286421    2570 scope.go:117] "RemoveContainer" containerID="4b838d16503cf17a7c69852fcc2da23222a4c4a9d453e51ea39577a29805877e"
Jul 29 04:44:23.286674 ip-10-0-135-138 kubenswrapper[2570]: E0729 04:44:23.286652    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b838d16503cf17a7c69852fcc2da23222a4c4a9d453e51ea39577a29805877e\": container with ID starting with 4b838d16503cf17a7c69852fcc2da23222a4c4a9d453e51ea39577a29805877e not found: ID does not exist" containerID="4b838d16503cf17a7c69852fcc2da23222a4c4a9d453e51ea39577a29805877e"
Jul 29 04:44:23.286748 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:23.286684    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b838d16503cf17a7c69852fcc2da23222a4c4a9d453e51ea39577a29805877e"} err="failed to get container status \"4b838d16503cf17a7c69852fcc2da23222a4c4a9d453e51ea39577a29805877e\": rpc error: code = NotFound desc = could not find container \"4b838d16503cf17a7c69852fcc2da23222a4c4a9d453e51ea39577a29805877e\": container with ID starting with 4b838d16503cf17a7c69852fcc2da23222a4c4a9d453e51ea39577a29805877e not found: ID does not exist"
Jul 29 04:44:23.286748 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:23.286704    2570 scope.go:117] "RemoveContainer" containerID="62f6b6c89f1638937c3bc7bc838dbca2354943700273799eb152719fc8800f47"
Jul 29 04:44:23.294071 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:23.294047    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fec70b-588bc697d8-62nvd"]
Jul 29 04:44:23.296142 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:23.296125    2570 scope.go:117] "RemoveContainer" containerID="62f6b6c89f1638937c3bc7bc838dbca2354943700273799eb152719fc8800f47"
Jul 29 04:44:23.296402 ip-10-0-135-138 kubenswrapper[2570]: E0729 04:44:23.296383    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62f6b6c89f1638937c3bc7bc838dbca2354943700273799eb152719fc8800f47\": container with ID starting with 62f6b6c89f1638937c3bc7bc838dbca2354943700273799eb152719fc8800f47 not found: ID does not exist" containerID="62f6b6c89f1638937c3bc7bc838dbca2354943700273799eb152719fc8800f47"
Jul 29 04:44:23.296468 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:23.296411    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62f6b6c89f1638937c3bc7bc838dbca2354943700273799eb152719fc8800f47"} err="failed to get container status \"62f6b6c89f1638937c3bc7bc838dbca2354943700273799eb152719fc8800f47\": rpc error: code = NotFound desc = could not find container \"62f6b6c89f1638937c3bc7bc838dbca2354943700273799eb152719fc8800f47\": container with ID starting with 62f6b6c89f1638937c3bc7bc838dbca2354943700273799eb152719fc8800f47 not found: ID does not exist"
Jul 29 04:44:23.297059 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:23.297043    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fec70b-588bc697d8-62nvd"]
Jul 29 04:44:23.306450 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:23.306431    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fec70b-5cb59fb-8bqgd"]
Jul 29 04:44:23.310013 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:23.309992    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fec70b-5cb59fb-8bqgd"]
Jul 29 04:44:23.320248 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:23.320229    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fec70b-5bd6f8467d-dpbfq"]
Jul 29 04:44:23.325191 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:23.325171    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fec70b-5bd6f8467d-dpbfq"]
Jul 29 04:44:23.834724 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:23.834691    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b61fc09-0d50-4853-adb9-caa01675f15f" path="/var/lib/kubelet/pods/4b61fc09-0d50-4853-adb9-caa01675f15f/volumes"
Jul 29 04:44:23.835109 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:23.835095    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c6faed0-01a8-4dea-bc4d-5bcae593f311" path="/var/lib/kubelet/pods/6c6faed0-01a8-4dea-bc4d-5bcae593f311/volumes"
Jul 29 04:44:23.835400 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:23.835388    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f8b902e-b00a-4b34-a796-2e62fabefcaf" path="/var/lib/kubelet/pods/8f8b902e-b00a-4b34-a796-2e62fabefcaf/volumes"
Jul 29 04:44:29.593717 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:29.593678    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a099ac-5cfjv"]
Jul 29 04:44:29.594331 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:29.594309    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b61fc09-0d50-4853-adb9-caa01675f15f" containerName="payload-processing"
Jul 29 04:44:29.594424 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:29.594336    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b61fc09-0d50-4853-adb9-caa01675f15f" containerName="payload-processing"
Jul 29 04:44:29.594424 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:29.594356    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c6faed0-01a8-4dea-bc4d-5bcae593f311" containerName="maas-api"
Jul 29 04:44:29.594424 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:29.594364    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c6faed0-01a8-4dea-bc4d-5bcae593f311" containerName="maas-api"
Jul 29 04:44:29.594424 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:29.594384    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f8b902e-b00a-4b34-a796-2e62fabefcaf" containerName="payload-pre-processing"
Jul 29 04:44:29.594424 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:29.594393    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f8b902e-b00a-4b34-a796-2e62fabefcaf" containerName="payload-pre-processing"
Jul 29 04:44:29.594666 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:29.594503    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b61fc09-0d50-4853-adb9-caa01675f15f" containerName="payload-processing"
Jul 29 04:44:29.594666 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:29.594515    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f8b902e-b00a-4b34-a796-2e62fabefcaf" containerName="payload-pre-processing"
Jul 29 04:44:29.594666 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:29.594530    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c6faed0-01a8-4dea-bc4d-5bcae593f311" containerName="maas-api"
Jul 29 04:44:29.599175 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:29.599152    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a099ac-5cfjv"
Jul 29 04:44:29.601872 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:29.601851    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8frm\""
Jul 29 04:44:29.605205 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:29.605186    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a099ac-5cfjv"]
Jul 29 04:44:29.674993 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:29.674965    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/888704a4-850e-4b20-b929-4d45c965831b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a099ac-5cfjv\" (UID: \"888704a4-850e-4b20-b929-4d45c965831b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a099ac-5cfjv"
Jul 29 04:44:29.776275 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:29.776239    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/888704a4-850e-4b20-b929-4d45c965831b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a099ac-5cfjv\" (UID: \"888704a4-850e-4b20-b929-4d45c965831b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a099ac-5cfjv"
Jul 29 04:44:29.776917 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:29.776897    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/888704a4-850e-4b20-b929-4d45c965831b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a099ac-5cfjv\" (UID: \"888704a4-850e-4b20-b929-4d45c965831b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a099ac-5cfjv"
Jul 29 04:44:29.911337 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:29.911258    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a099ac-5cfjv"
Jul 29 04:44:30.047224 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:30.047197    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a099ac-5cfjv"]
Jul 29 04:44:30.049555 ip-10-0-135-138 kubenswrapper[2570]: W0729 04:44:30.049528    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod888704a4_850e_4b20_b929_4d45c965831b.slice/crio-dd98a8184f295183b16fd6d76c3b72cc4918c0221de3cafc51594f1f541126d9 WatchSource:0}: Error finding container dd98a8184f295183b16fd6d76c3b72cc4918c0221de3cafc51594f1f541126d9: Status 404 returned error can't find the container with id dd98a8184f295183b16fd6d76c3b72cc4918c0221de3cafc51594f1f541126d9
Jul 29 04:44:30.307172 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:30.307140    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a099ac-5cfjv" event={"ID":"888704a4-850e-4b20-b929-4d45c965831b","Type":"ContainerStarted","Data":"dd98a8184f295183b16fd6d76c3b72cc4918c0221de3cafc51594f1f541126d9"}
Jul 29 04:44:31.313765 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:31.313733    2570 generic.go:356] "Generic (PLEG): container finished" podID="888704a4-850e-4b20-b929-4d45c965831b" containerID="e437132bde0ec9e01e868bf5610cc537d94301f0b14b69780765de20f8759bdd" exitCode=0
Jul 29 04:44:31.314225 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:31.313834    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a099ac-5cfjv" event={"ID":"888704a4-850e-4b20-b929-4d45c965831b","Type":"ContainerDied","Data":"e437132bde0ec9e01e868bf5610cc537d94301f0b14b69780765de20f8759bdd"}
Jul 29 04:44:32.449097 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:32.449072    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a099ac-5cfjv"
Jul 29 04:44:32.501921 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:32.501888    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/888704a4-850e-4b20-b929-4d45c965831b-maas-api-service-ca\") pod \"888704a4-850e-4b20-b929-4d45c965831b\" (UID: \"888704a4-850e-4b20-b929-4d45c965831b\") "
Jul 29 04:44:32.502208 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:32.502187    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/888704a4-850e-4b20-b929-4d45c965831b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "888704a4-850e-4b20-b929-4d45c965831b" (UID: "888704a4-850e-4b20-b929-4d45c965831b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 04:44:32.603357 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:32.603334    2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/888704a4-850e-4b20-b929-4d45c965831b-maas-api-service-ca\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:44:33.325477 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:33.325451    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a099ac-5cfjv"
Jul 29 04:44:33.325648 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:33.325490    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a099ac-5cfjv" event={"ID":"888704a4-850e-4b20-b929-4d45c965831b","Type":"ContainerDied","Data":"dd98a8184f295183b16fd6d76c3b72cc4918c0221de3cafc51594f1f541126d9"}
Jul 29 04:44:33.325648 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:33.325542    2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd98a8184f295183b16fd6d76c3b72cc4918c0221de3cafc51594f1f541126d9"
Jul 29 04:44:45.326356 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:45.326315    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a099ac-64c95c68d9-hghgt"]
Jul 29 04:44:45.326920 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:45.326568    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a099ac-64c95c68d9-hghgt" podUID="abe81f5f-dae4-4b5c-8a5b-9cccc5296213" containerName="maas-api" containerID="cri-o://035650d65d751054453efb4470be9bf30d420d9fdc422be0ca432142068f4f27" gracePeriod=30
Jul 29 04:44:45.393182 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:45.393131    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a099ac-658c674bcb-72skk"]
Jul 29 04:44:45.393436 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:45.393401    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a099ac-658c674bcb-72skk" podUID="595da500-eb7b-4a3f-aac7-00d328153825" containerName="payload-pre-processing" containerID="cri-o://ca11f4f8f89ffaba0c35c46a03704276b87df02cb5dea5947baa2140194eed1f" gracePeriod=30
Jul 29 04:44:45.500195 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:45.500161    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a099ac-5cfjv"]
Jul 29 04:44:45.501873 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:45.501851    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a099ac-5cfjv"]
Jul 29 04:44:45.666375 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:45.666353    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a099ac-64c95c68d9-hghgt"
Jul 29 04:44:45.669516 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:45.669487    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a099ac-658c674bcb-72skk"
Jul 29 04:44:45.824353 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:45.824323    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-frtgv\" (UniqueName: \"kubernetes.io/projected/abe81f5f-dae4-4b5c-8a5b-9cccc5296213-kube-api-access-frtgv\") pod \"abe81f5f-dae4-4b5c-8a5b-9cccc5296213\" (UID: \"abe81f5f-dae4-4b5c-8a5b-9cccc5296213\") "
Jul 29 04:44:45.824353 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:45.824357    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abe81f5f-dae4-4b5c-8a5b-9cccc5296213-openshift-service-ca\") pod \"abe81f5f-dae4-4b5c-8a5b-9cccc5296213\" (UID: \"abe81f5f-dae4-4b5c-8a5b-9cccc5296213\") "
Jul 29 04:44:45.824595 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:45.824383    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7hjxm\" (UniqueName: \"kubernetes.io/projected/595da500-eb7b-4a3f-aac7-00d328153825-kube-api-access-7hjxm\") pod \"595da500-eb7b-4a3f-aac7-00d328153825\" (UID: \"595da500-eb7b-4a3f-aac7-00d328153825\") "
Jul 29 04:44:45.824595 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:45.824521    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abe81f5f-dae4-4b5c-8a5b-9cccc5296213-maas-api-tls\") pod \"abe81f5f-dae4-4b5c-8a5b-9cccc5296213\" (UID: \"abe81f5f-dae4-4b5c-8a5b-9cccc5296213\") "
Jul 29 04:44:45.824595 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:45.824572    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/595da500-eb7b-4a3f-aac7-00d328153825-plugins-config-volume\") pod \"595da500-eb7b-4a3f-aac7-00d328153825\" (UID: \"595da500-eb7b-4a3f-aac7-00d328153825\") "
Jul 29 04:44:45.824788 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:45.824721    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abe81f5f-dae4-4b5c-8a5b-9cccc5296213-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "abe81f5f-dae4-4b5c-8a5b-9cccc5296213" (UID: "abe81f5f-dae4-4b5c-8a5b-9cccc5296213"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 04:44:45.825064 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:45.825044    2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abe81f5f-dae4-4b5c-8a5b-9cccc5296213-openshift-service-ca\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:44:45.825358 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:45.825075    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/595da500-eb7b-4a3f-aac7-00d328153825-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "595da500-eb7b-4a3f-aac7-00d328153825" (UID: "595da500-eb7b-4a3f-aac7-00d328153825"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 04:44:45.826860 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:45.826810    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abe81f5f-dae4-4b5c-8a5b-9cccc5296213-kube-api-access-frtgv" (OuterVolumeSpecName: "kube-api-access-frtgv") pod "abe81f5f-dae4-4b5c-8a5b-9cccc5296213" (UID: "abe81f5f-dae4-4b5c-8a5b-9cccc5296213"). InnerVolumeSpecName "kube-api-access-frtgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 04:44:45.826971 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:45.826928    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abe81f5f-dae4-4b5c-8a5b-9cccc5296213-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "abe81f5f-dae4-4b5c-8a5b-9cccc5296213" (UID: "abe81f5f-dae4-4b5c-8a5b-9cccc5296213"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 04:44:45.826971 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:45.826956    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/595da500-eb7b-4a3f-aac7-00d328153825-kube-api-access-7hjxm" (OuterVolumeSpecName: "kube-api-access-7hjxm") pod "595da500-eb7b-4a3f-aac7-00d328153825" (UID: "595da500-eb7b-4a3f-aac7-00d328153825"). InnerVolumeSpecName "kube-api-access-7hjxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 04:44:45.835753 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:45.835724    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="888704a4-850e-4b20-b929-4d45c965831b" path="/var/lib/kubelet/pods/888704a4-850e-4b20-b929-4d45c965831b/volumes"
Jul 29 04:44:45.925580 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:45.925543    2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/595da500-eb7b-4a3f-aac7-00d328153825-plugins-config-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:44:45.925580 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:45.925575    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frtgv\" (UniqueName: \"kubernetes.io/projected/abe81f5f-dae4-4b5c-8a5b-9cccc5296213-kube-api-access-frtgv\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:44:45.925580 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:45.925586    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hjxm\" (UniqueName: \"kubernetes.io/projected/595da500-eb7b-4a3f-aac7-00d328153825-kube-api-access-7hjxm\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:44:45.925798 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:45.925595    2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abe81f5f-dae4-4b5c-8a5b-9cccc5296213-maas-api-tls\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:44:46.402937 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:46.402900    2570 generic.go:356] "Generic (PLEG): container finished" podID="abe81f5f-dae4-4b5c-8a5b-9cccc5296213" containerID="035650d65d751054453efb4470be9bf30d420d9fdc422be0ca432142068f4f27" exitCode=0
Jul 29 04:44:46.403322 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:46.402945    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a099ac-64c95c68d9-hghgt"
Jul 29 04:44:46.403322 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:46.402986    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a099ac-64c95c68d9-hghgt" event={"ID":"abe81f5f-dae4-4b5c-8a5b-9cccc5296213","Type":"ContainerDied","Data":"035650d65d751054453efb4470be9bf30d420d9fdc422be0ca432142068f4f27"}
Jul 29 04:44:46.403322 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:46.403041    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a099ac-64c95c68d9-hghgt" event={"ID":"abe81f5f-dae4-4b5c-8a5b-9cccc5296213","Type":"ContainerDied","Data":"ae425993586ffb3e19ab2f10c7f26318faf1513d205ec4dbb77f0cec3225b326"}
Jul 29 04:44:46.403322 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:46.403067    2570 scope.go:117] "RemoveContainer" containerID="035650d65d751054453efb4470be9bf30d420d9fdc422be0ca432142068f4f27"
Jul 29 04:44:46.404703 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:46.404679    2570 generic.go:356] "Generic (PLEG): container finished" podID="595da500-eb7b-4a3f-aac7-00d328153825" containerID="ca11f4f8f89ffaba0c35c46a03704276b87df02cb5dea5947baa2140194eed1f" exitCode=0
Jul 29 04:44:46.404839 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:46.404739    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a099ac-658c674bcb-72skk" event={"ID":"595da500-eb7b-4a3f-aac7-00d328153825","Type":"ContainerDied","Data":"ca11f4f8f89ffaba0c35c46a03704276b87df02cb5dea5947baa2140194eed1f"}
Jul 29 04:44:46.404839 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:46.404783    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a099ac-658c674bcb-72skk" event={"ID":"595da500-eb7b-4a3f-aac7-00d328153825","Type":"ContainerDied","Data":"649c6026ebc1553dafd6c6698f5dda5099c44dff3b91d696fd8711e510e75ef0"}
Jul 29 04:44:46.404965 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:46.404873    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a099ac-658c674bcb-72skk"
Jul 29 04:44:46.412129 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:46.412109    2570 scope.go:117] "RemoveContainer" containerID="035650d65d751054453efb4470be9bf30d420d9fdc422be0ca432142068f4f27"
Jul 29 04:44:46.412376 ip-10-0-135-138 kubenswrapper[2570]: E0729 04:44:46.412350    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"035650d65d751054453efb4470be9bf30d420d9fdc422be0ca432142068f4f27\": container with ID starting with 035650d65d751054453efb4470be9bf30d420d9fdc422be0ca432142068f4f27 not found: ID does not exist" containerID="035650d65d751054453efb4470be9bf30d420d9fdc422be0ca432142068f4f27"
Jul 29 04:44:46.412465 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:46.412385    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"035650d65d751054453efb4470be9bf30d420d9fdc422be0ca432142068f4f27"} err="failed to get container status \"035650d65d751054453efb4470be9bf30d420d9fdc422be0ca432142068f4f27\": rpc error: code = NotFound desc = could not find container \"035650d65d751054453efb4470be9bf30d420d9fdc422be0ca432142068f4f27\": container with ID starting with 035650d65d751054453efb4470be9bf30d420d9fdc422be0ca432142068f4f27 not found: ID does not exist"
Jul 29 04:44:46.412465 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:46.412410    2570 scope.go:117] "RemoveContainer" containerID="ca11f4f8f89ffaba0c35c46a03704276b87df02cb5dea5947baa2140194eed1f"
Jul 29 04:44:46.420498 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:46.420481    2570 scope.go:117] "RemoveContainer" containerID="ca11f4f8f89ffaba0c35c46a03704276b87df02cb5dea5947baa2140194eed1f"
Jul 29 04:44:46.420781 ip-10-0-135-138 kubenswrapper[2570]: E0729 04:44:46.420751    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca11f4f8f89ffaba0c35c46a03704276b87df02cb5dea5947baa2140194eed1f\": container with ID starting with ca11f4f8f89ffaba0c35c46a03704276b87df02cb5dea5947baa2140194eed1f not found: ID does not exist" containerID="ca11f4f8f89ffaba0c35c46a03704276b87df02cb5dea5947baa2140194eed1f"
Jul 29 04:44:46.420912 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:46.420779    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca11f4f8f89ffaba0c35c46a03704276b87df02cb5dea5947baa2140194eed1f"} err="failed to get container status \"ca11f4f8f89ffaba0c35c46a03704276b87df02cb5dea5947baa2140194eed1f\": rpc error: code = NotFound desc = could not find container \"ca11f4f8f89ffaba0c35c46a03704276b87df02cb5dea5947baa2140194eed1f\": container with ID starting with ca11f4f8f89ffaba0c35c46a03704276b87df02cb5dea5947baa2140194eed1f not found: ID does not exist"
Jul 29 04:44:46.423895 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:46.423874    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a099ac-658c674bcb-72skk"]
Jul 29 04:44:46.425595 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:46.425571    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a099ac-658c674bcb-72skk"]
Jul 29 04:44:46.435696 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:46.435676    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a099ac-64c95c68d9-hghgt"]
Jul 29 04:44:46.437269 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:46.437239    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a099ac-64c95c68d9-hghgt"]
Jul 29 04:44:47.835149 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:47.835122    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="595da500-eb7b-4a3f-aac7-00d328153825" path="/var/lib/kubelet/pods/595da500-eb7b-4a3f-aac7-00d328153825/volumes"
Jul 29 04:44:47.835512 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:47.835477    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abe81f5f-dae4-4b5c-8a5b-9cccc5296213" path="/var/lib/kubelet/pods/abe81f5f-dae4-4b5c-8a5b-9cccc5296213/volumes"
Jul 29 04:44:55.802728 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:55.802696    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-b70f7c-c9dd4b449-xhff9"]
Jul 29 04:44:55.803183 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:55.803077    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="595da500-eb7b-4a3f-aac7-00d328153825" containerName="payload-pre-processing"
Jul 29 04:44:55.803183 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:55.803088    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="595da500-eb7b-4a3f-aac7-00d328153825" containerName="payload-pre-processing"
Jul 29 04:44:55.803183 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:55.803106    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="888704a4-850e-4b20-b929-4d45c965831b" containerName="revoke-keys"
Jul 29 04:44:55.803183 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:55.803111    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="888704a4-850e-4b20-b929-4d45c965831b" containerName="revoke-keys"
Jul 29 04:44:55.803183 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:55.803122    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abe81f5f-dae4-4b5c-8a5b-9cccc5296213" containerName="maas-api"
Jul 29 04:44:55.803183 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:55.803128    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="abe81f5f-dae4-4b5c-8a5b-9cccc5296213" containerName="maas-api"
Jul 29 04:44:55.803183 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:55.803183    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="595da500-eb7b-4a3f-aac7-00d328153825" containerName="payload-pre-processing"
Jul 29 04:44:55.803404 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:55.803192    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="abe81f5f-dae4-4b5c-8a5b-9cccc5296213" containerName="maas-api"
Jul 29 04:44:55.803404 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:55.803203    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="888704a4-850e-4b20-b929-4d45c965831b" containerName="revoke-keys"
Jul 29 04:44:55.807691 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:55.807668    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b70f7c-c9dd4b449-xhff9"
Jul 29 04:44:55.810096 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:55.810070    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-lh6p9\""
Jul 29 04:44:55.810223 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:55.810076    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-b70f7c-serving-cert\""
Jul 29 04:44:55.812064 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:55.812035    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 29 04:44:55.812447 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:55.812420    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 29 04:44:55.820012 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:55.819578    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-b70f7c-c9dd4b449-xhff9"]
Jul 29 04:44:55.919846 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:55.919785    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69c5c2d6-a4dc-4108-91f5-e1fb2dd7234b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-b70f7c-c9dd4b449-xhff9\" (UID: \"69c5c2d6-a4dc-4108-91f5-e1fb2dd7234b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b70f7c-c9dd4b449-xhff9"
Jul 29 04:44:55.920021 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:55.919918    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69c5c2d6-a4dc-4108-91f5-e1fb2dd7234b-maas-api-tls\") pod \"maas-api-e2e-shared-a-b70f7c-c9dd4b449-xhff9\" (UID: \"69c5c2d6-a4dc-4108-91f5-e1fb2dd7234b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b70f7c-c9dd4b449-xhff9"
Jul 29 04:44:55.920021 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:55.920004    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmmpf\" (UniqueName: \"kubernetes.io/projected/69c5c2d6-a4dc-4108-91f5-e1fb2dd7234b-kube-api-access-vmmpf\") pod \"maas-api-e2e-shared-a-b70f7c-c9dd4b449-xhff9\" (UID: \"69c5c2d6-a4dc-4108-91f5-e1fb2dd7234b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b70f7c-c9dd4b449-xhff9"
Jul 29 04:44:56.021340 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:56.021308    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69c5c2d6-a4dc-4108-91f5-e1fb2dd7234b-maas-api-tls\") pod \"maas-api-e2e-shared-a-b70f7c-c9dd4b449-xhff9\" (UID: \"69c5c2d6-a4dc-4108-91f5-e1fb2dd7234b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b70f7c-c9dd4b449-xhff9"
Jul 29 04:44:56.021496 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:56.021365    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmmpf\" (UniqueName: \"kubernetes.io/projected/69c5c2d6-a4dc-4108-91f5-e1fb2dd7234b-kube-api-access-vmmpf\") pod \"maas-api-e2e-shared-a-b70f7c-c9dd4b449-xhff9\" (UID: \"69c5c2d6-a4dc-4108-91f5-e1fb2dd7234b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b70f7c-c9dd4b449-xhff9"
Jul 29 04:44:56.021496 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:56.021466    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69c5c2d6-a4dc-4108-91f5-e1fb2dd7234b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-b70f7c-c9dd4b449-xhff9\" (UID: \"69c5c2d6-a4dc-4108-91f5-e1fb2dd7234b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b70f7c-c9dd4b449-xhff9"
Jul 29 04:44:56.022046 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:56.022024    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69c5c2d6-a4dc-4108-91f5-e1fb2dd7234b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-b70f7c-c9dd4b449-xhff9\" (UID: \"69c5c2d6-a4dc-4108-91f5-e1fb2dd7234b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b70f7c-c9dd4b449-xhff9"
Jul 29 04:44:56.023808 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:56.023789    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69c5c2d6-a4dc-4108-91f5-e1fb2dd7234b-maas-api-tls\") pod \"maas-api-e2e-shared-a-b70f7c-c9dd4b449-xhff9\" (UID: \"69c5c2d6-a4dc-4108-91f5-e1fb2dd7234b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b70f7c-c9dd4b449-xhff9"
Jul 29 04:44:56.028856 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:56.028804    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmmpf\" (UniqueName: \"kubernetes.io/projected/69c5c2d6-a4dc-4108-91f5-e1fb2dd7234b-kube-api-access-vmmpf\") pod \"maas-api-e2e-shared-a-b70f7c-c9dd4b449-xhff9\" (UID: \"69c5c2d6-a4dc-4108-91f5-e1fb2dd7234b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b70f7c-c9dd4b449-xhff9"
Jul 29 04:44:56.124561 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:56.124501    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b70f7c-c9dd4b449-xhff9"
Jul 29 04:44:56.255690 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:56.255664    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-b70f7c-c9dd4b449-xhff9"]
Jul 29 04:44:56.258455 ip-10-0-135-138 kubenswrapper[2570]: W0729 04:44:56.258429    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69c5c2d6_a4dc_4108_91f5_e1fb2dd7234b.slice/crio-e11cd7804322ea79f5ef62139455eafb2aaf5857058c9359d9b483f3fa0d0ebc WatchSource:0}: Error finding container e11cd7804322ea79f5ef62139455eafb2aaf5857058c9359d9b483f3fa0d0ebc: Status 404 returned error can't find the container with id e11cd7804322ea79f5ef62139455eafb2aaf5857058c9359d9b483f3fa0d0ebc
Jul 29 04:44:56.460639 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:56.460554    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b70f7c-c9dd4b449-xhff9" event={"ID":"69c5c2d6-a4dc-4108-91f5-e1fb2dd7234b","Type":"ContainerStarted","Data":"e11cd7804322ea79f5ef62139455eafb2aaf5857058c9359d9b483f3fa0d0ebc"}
Jul 29 04:44:56.579673 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:56.579644    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-b70f7c-f59c576c6-5p9q7"]
Jul 29 04:44:56.584389 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:56.584373    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-b70f7c-f59c576c6-5p9q7"
Jul 29 04:44:56.586831 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:56.586787    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-b70f7c\""
Jul 29 04:44:56.586831 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:56.586809    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-b70f7c-dockercfg-7vzw6\""
Jul 29 04:44:56.594344 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:56.594324    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-b70f7c-f59c576c6-5p9q7"]
Jul 29 04:44:56.609788 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:56.609750    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-b70f7c-5f8c686d6b-7c6xx"]
Jul 29 04:44:56.613069 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:56.613042    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-b70f7c-5f8c686d6b-7c6xx"
Jul 29 04:44:56.624811 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:56.624645    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-b70f7c-5f8c686d6b-7c6xx"]
Jul 29 04:44:56.727613 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:56.727548    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hpft\" (UniqueName: \"kubernetes.io/projected/57aa4758-0757-4c92-b9c6-d3eb724a6a37-kube-api-access-4hpft\") pod \"payload-pre-processing-e2e-shared-a-b70f7c-f59c576c6-5p9q7\" (UID: \"57aa4758-0757-4c92-b9c6-d3eb724a6a37\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-b70f7c-f59c576c6-5p9q7"
Jul 29 04:44:56.727613 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:56.727601    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzf2n\" (UniqueName: \"kubernetes.io/projected/f719bfad-7662-4706-aed4-c080a4912db2-kube-api-access-jzf2n\") pod \"payload-processing-e2e-shared-a-b70f7c-5f8c686d6b-7c6xx\" (UID: \"f719bfad-7662-4706-aed4-c080a4912db2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-b70f7c-5f8c686d6b-7c6xx"
Jul 29 04:44:56.727780 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:56.727657    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f719bfad-7662-4706-aed4-c080a4912db2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-b70f7c-5f8c686d6b-7c6xx\" (UID: \"f719bfad-7662-4706-aed4-c080a4912db2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-b70f7c-5f8c686d6b-7c6xx"
Jul 29 04:44:56.727780 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:56.727699    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57aa4758-0757-4c92-b9c6-d3eb724a6a37-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-b70f7c-f59c576c6-5p9q7\" (UID: \"57aa4758-0757-4c92-b9c6-d3eb724a6a37\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-b70f7c-f59c576c6-5p9q7"
Jul 29 04:44:56.828992 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:56.828964    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f719bfad-7662-4706-aed4-c080a4912db2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-b70f7c-5f8c686d6b-7c6xx\" (UID: \"f719bfad-7662-4706-aed4-c080a4912db2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-b70f7c-5f8c686d6b-7c6xx"
Jul 29 04:44:56.829424 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:56.829000    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57aa4758-0757-4c92-b9c6-d3eb724a6a37-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-b70f7c-f59c576c6-5p9q7\" (UID: \"57aa4758-0757-4c92-b9c6-d3eb724a6a37\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-b70f7c-f59c576c6-5p9q7"
Jul 29 04:44:56.829424 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:56.829078    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hpft\" (UniqueName: \"kubernetes.io/projected/57aa4758-0757-4c92-b9c6-d3eb724a6a37-kube-api-access-4hpft\") pod \"payload-pre-processing-e2e-shared-a-b70f7c-f59c576c6-5p9q7\" (UID: \"57aa4758-0757-4c92-b9c6-d3eb724a6a37\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-b70f7c-f59c576c6-5p9q7"
Jul 29 04:44:56.829424 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:56.829356    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzf2n\" (UniqueName: \"kubernetes.io/projected/f719bfad-7662-4706-aed4-c080a4912db2-kube-api-access-jzf2n\") pod \"payload-processing-e2e-shared-a-b70f7c-5f8c686d6b-7c6xx\" (UID: \"f719bfad-7662-4706-aed4-c080a4912db2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-b70f7c-5f8c686d6b-7c6xx"
Jul 29 04:44:56.829787 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:56.829767    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f719bfad-7662-4706-aed4-c080a4912db2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-b70f7c-5f8c686d6b-7c6xx\" (UID: \"f719bfad-7662-4706-aed4-c080a4912db2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-b70f7c-5f8c686d6b-7c6xx"
Jul 29 04:44:56.829909 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:56.829894    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57aa4758-0757-4c92-b9c6-d3eb724a6a37-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-b70f7c-f59c576c6-5p9q7\" (UID: \"57aa4758-0757-4c92-b9c6-d3eb724a6a37\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-b70f7c-f59c576c6-5p9q7"
Jul 29 04:44:56.837665 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:56.837631    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hpft\" (UniqueName: \"kubernetes.io/projected/57aa4758-0757-4c92-b9c6-d3eb724a6a37-kube-api-access-4hpft\") pod \"payload-pre-processing-e2e-shared-a-b70f7c-f59c576c6-5p9q7\" (UID: \"57aa4758-0757-4c92-b9c6-d3eb724a6a37\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-b70f7c-f59c576c6-5p9q7"
Jul 29 04:44:56.838153 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:56.838132    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzf2n\" (UniqueName: \"kubernetes.io/projected/f719bfad-7662-4706-aed4-c080a4912db2-kube-api-access-jzf2n\") pod \"payload-processing-e2e-shared-a-b70f7c-5f8c686d6b-7c6xx\" (UID: \"f719bfad-7662-4706-aed4-c080a4912db2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-b70f7c-5f8c686d6b-7c6xx"
Jul 29 04:44:56.896942 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:56.896917    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-b70f7c-f59c576c6-5p9q7"
Jul 29 04:44:56.927691 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:56.927668    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-b70f7c-5f8c686d6b-7c6xx"
Jul 29 04:44:57.041189 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:57.041160    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-b70f7c-f59c576c6-5p9q7"]
Jul 29 04:44:57.045091 ip-10-0-135-138 kubenswrapper[2570]: W0729 04:44:57.045057    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57aa4758_0757_4c92_b9c6_d3eb724a6a37.slice/crio-563f5cd0482a1b69b79d92cfc4c6e668934e0d4bc6d7110b71d86ce6af1c965a WatchSource:0}: Error finding container 563f5cd0482a1b69b79d92cfc4c6e668934e0d4bc6d7110b71d86ce6af1c965a: Status 404 returned error can't find the container with id 563f5cd0482a1b69b79d92cfc4c6e668934e0d4bc6d7110b71d86ce6af1c965a
Jul 29 04:44:57.081507 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:57.081473    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-b70f7c-5f8c686d6b-7c6xx"]
Jul 29 04:44:57.084009 ip-10-0-135-138 kubenswrapper[2570]: W0729 04:44:57.083974    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf719bfad_7662_4706_aed4_c080a4912db2.slice/crio-878d8c64fa34ec0d982617f113dda5d29c2dfeae4564f7dcc92d8ab135eb1b12 WatchSource:0}: Error finding container 878d8c64fa34ec0d982617f113dda5d29c2dfeae4564f7dcc92d8ab135eb1b12: Status 404 returned error can't find the container with id 878d8c64fa34ec0d982617f113dda5d29c2dfeae4564f7dcc92d8ab135eb1b12
Jul 29 04:44:57.467462 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:57.467423    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b70f7c-c9dd4b449-xhff9" event={"ID":"69c5c2d6-a4dc-4108-91f5-e1fb2dd7234b","Type":"ContainerStarted","Data":"dc8ead7183969f65ea8cc10c5979c158b35eee34dff43df807daae56d2d5a86e"}
Jul 29 04:44:57.467651 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:57.467487    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b70f7c-c9dd4b449-xhff9"
Jul 29 04:44:57.468949 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:57.468923    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-b70f7c-5f8c686d6b-7c6xx" event={"ID":"f719bfad-7662-4706-aed4-c080a4912db2","Type":"ContainerStarted","Data":"ff8556ea9c959699ac11804b8357ca74c6b66ea198af498aa54a76c825bcef30"}
Jul 29 04:44:57.469070 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:57.468964    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-b70f7c-5f8c686d6b-7c6xx" event={"ID":"f719bfad-7662-4706-aed4-c080a4912db2","Type":"ContainerStarted","Data":"878d8c64fa34ec0d982617f113dda5d29c2dfeae4564f7dcc92d8ab135eb1b12"}
Jul 29 04:44:57.469070 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:57.469026    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-b70f7c-5f8c686d6b-7c6xx"
Jul 29 04:44:57.470322 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:57.470301    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-b70f7c-f59c576c6-5p9q7" event={"ID":"57aa4758-0757-4c92-b9c6-d3eb724a6a37","Type":"ContainerStarted","Data":"62838deb44f595c26ef8ca4d494eb77eab4963c0c671436f28119dffe8d57f01"}
Jul 29 04:44:57.470411 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:57.470342    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-b70f7c-f59c576c6-5p9q7" event={"ID":"57aa4758-0757-4c92-b9c6-d3eb724a6a37","Type":"ContainerStarted","Data":"563f5cd0482a1b69b79d92cfc4c6e668934e0d4bc6d7110b71d86ce6af1c965a"}
Jul 29 04:44:57.470411 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:57.470361    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-b70f7c-f59c576c6-5p9q7"
Jul 29 04:44:57.482615 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:57.482575    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b70f7c-c9dd4b449-xhff9" podStartSLOduration=2.12033674 podStartE2EDuration="2.482560957s" totalImagesPullingTime="362.224217ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:44:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:44:57.481891087 +0000 UTC m=+2962.212022183" watchObservedRunningTime="2026-07-29 04:44:57.482560957 +0000 UTC m=+2962.212692026"
Jul 29 04:44:57.496396 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:57.496356    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-b70f7c-f59c576c6-5p9q7" podStartSLOduration=1.4963440399999999 podStartE2EDuration="1.49634404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:44:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:44:57.495647566 +0000 UTC m=+2962.225778675" watchObservedRunningTime="2026-07-29 04:44:57.49634404 +0000 UTC m=+2962.226475110"
Jul 29 04:44:57.512425 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:44:57.512378    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-b70f7c-5f8c686d6b-7c6xx" podStartSLOduration=1.512360286 podStartE2EDuration="1.512360286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:44:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:44:57.510274616 +0000 UTC m=+2962.240405708" watchObservedRunningTime="2026-07-29 04:44:57.512360286 +0000 UTC m=+2962.242491362"
Jul 29 04:45:03.486872 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:45:03.486806    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b70f7c-c9dd4b449-xhff9"
Jul 29 04:45:08.483466 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:45:08.483432    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-b70f7c-5f8c686d6b-7c6xx"
Jul 29 04:45:08.483884 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:45:08.483507    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-b70f7c-f59c576c6-5p9q7"
Jul 29 04:45:24.832493 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:45:24.832458    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-b550f6-6c8767bd9b-z4lkb"]
Jul 29 04:45:24.836026 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:45:24.836010    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-b550f6-6c8767bd9b-z4lkb"
Jul 29 04:45:24.838561 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:45:24.838539    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-b550f6-dockercfg-fpcrl\""
Jul 29 04:45:24.838682 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:45:24.838563    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-b550f6\""
Jul 29 04:45:24.846016 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:45:24.845983    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-b550f6-6c8767bd9b-z4lkb"]
Jul 29 04:45:24.991590 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:45:24.991552    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e7e436c-94f3-47fb-b155-9ec558a4ecd1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-b550f6-6c8767bd9b-z4lkb\" (UID: \"3e7e436c-94f3-47fb-b155-9ec558a4ecd1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-b550f6-6c8767bd9b-z4lkb"
Jul 29 04:45:24.991786 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:45:24.991597    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2bc9\" (UniqueName: \"kubernetes.io/projected/3e7e436c-94f3-47fb-b155-9ec558a4ecd1-kube-api-access-n2bc9\") pod \"payload-pre-processing-e2e-shared-b-b550f6-6c8767bd9b-z4lkb\" (UID: \"3e7e436c-94f3-47fb-b155-9ec558a4ecd1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-b550f6-6c8767bd9b-z4lkb"
Jul 29 04:45:25.092676 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:45:25.092591    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e7e436c-94f3-47fb-b155-9ec558a4ecd1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-b550f6-6c8767bd9b-z4lkb\" (UID: \"3e7e436c-94f3-47fb-b155-9ec558a4ecd1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-b550f6-6c8767bd9b-z4lkb"
Jul 29 04:45:25.092676 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:45:25.092631    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2bc9\" (UniqueName: \"kubernetes.io/projected/3e7e436c-94f3-47fb-b155-9ec558a4ecd1-kube-api-access-n2bc9\") pod \"payload-pre-processing-e2e-shared-b-b550f6-6c8767bd9b-z4lkb\" (UID: \"3e7e436c-94f3-47fb-b155-9ec558a4ecd1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-b550f6-6c8767bd9b-z4lkb"
Jul 29 04:45:25.093392 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:45:25.093373    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e7e436c-94f3-47fb-b155-9ec558a4ecd1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-b550f6-6c8767bd9b-z4lkb\" (UID: \"3e7e436c-94f3-47fb-b155-9ec558a4ecd1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-b550f6-6c8767bd9b-z4lkb"
Jul 29 04:45:25.107007 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:45:25.106977    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2bc9\" (UniqueName: \"kubernetes.io/projected/3e7e436c-94f3-47fb-b155-9ec558a4ecd1-kube-api-access-n2bc9\") pod \"payload-pre-processing-e2e-shared-b-b550f6-6c8767bd9b-z4lkb\" (UID: \"3e7e436c-94f3-47fb-b155-9ec558a4ecd1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-b550f6-6c8767bd9b-z4lkb"
Jul 29 04:45:25.148106 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:45:25.148080    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-b550f6-6c8767bd9b-z4lkb"
Jul 29 04:45:25.283435 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:45:25.283407    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-b550f6-6c8767bd9b-z4lkb"]
Jul 29 04:45:25.285183 ip-10-0-135-138 kubenswrapper[2570]: W0729 04:45:25.285154    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e7e436c_94f3_47fb_b155_9ec558a4ecd1.slice/crio-3fcc64939cd9c886b10735ca6f263fd029bfe2825d41195a8159551d5dc2e4be WatchSource:0}: Error finding container 3fcc64939cd9c886b10735ca6f263fd029bfe2825d41195a8159551d5dc2e4be: Status 404 returned error can't find the container with id 3fcc64939cd9c886b10735ca6f263fd029bfe2825d41195a8159551d5dc2e4be
Jul 29 04:45:25.626172 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:45:25.626092    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-b550f6-6c8767bd9b-z4lkb" event={"ID":"3e7e436c-94f3-47fb-b155-9ec558a4ecd1","Type":"ContainerStarted","Data":"328dd955241340e54759d170034900538e5947bef565e0e46f2e544a97ec3c8f"}
Jul 29 04:45:25.626172 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:45:25.626154    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-b550f6-6c8767bd9b-z4lkb"
Jul 29 04:45:25.626172 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:45:25.626172    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-b550f6-6c8767bd9b-z4lkb" event={"ID":"3e7e436c-94f3-47fb-b155-9ec558a4ecd1","Type":"ContainerStarted","Data":"3fcc64939cd9c886b10735ca6f263fd029bfe2825d41195a8159551d5dc2e4be"}
Jul 29 04:45:25.646045 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:45:25.645997    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-b550f6-6c8767bd9b-z4lkb" podStartSLOduration=1.6459824570000001 podStartE2EDuration="1.645982457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:45:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:45:25.644032341 +0000 UTC m=+2990.374163445" watchObservedRunningTime="2026-07-29 04:45:25.645982457 +0000 UTC m=+2990.376113527"
Jul 29 04:45:36.633670 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:45:36.633636    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-b550f6-6c8767bd9b-z4lkb"
Jul 29 04:46:03.329115 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:46:03.329082    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-b70f7c/auth-test-model-b70f7c-kserve-7cd6d59d59-s9wps"]
Jul 29 04:46:03.332653 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:46:03.332632    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b70f7c/auth-test-model-b70f7c-kserve-7cd6d59d59-s9wps"
Jul 29 04:46:03.335660 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:46:03.335639    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-b70f7c\"/\"kube-root-ca.crt\""
Jul 29 04:46:03.337159 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:46:03.337139    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-b70f7c\"/\"default-dockercfg-4rfcc\""
Jul 29 04:46:03.337280 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:46:03.337141    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-b70f7c\"/\"auth-test-model-b70f7c-kserve-self-signed-certs\""
Jul 29 04:46:03.337280 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:46:03.337204    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-b70f7c\"/\"openshift-service-ca.crt\""
Jul 29 04:46:03.341949 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:46:03.341928    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-b70f7c/auth-test-model-b70f7c-kserve-7cd6d59d59-s9wps"]
Jul 29 04:46:03.447205 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:46:03.447167    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c9174a6-1dc9-44a0-abc8-21798694cf3d-dshm\") pod \"auth-test-model-b70f7c-kserve-7cd6d59d59-s9wps\" (UID: \"3c9174a6-1dc9-44a0-abc8-21798694cf3d\") " pod="ai-tenant-e2e-shared-a-b70f7c/auth-test-model-b70f7c-kserve-7cd6d59d59-s9wps"
Jul 29 04:46:03.447205 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:46:03.447208    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrqvq\" (UniqueName: \"kubernetes.io/projected/3c9174a6-1dc9-44a0-abc8-21798694cf3d-kube-api-access-jrqvq\") pod \"auth-test-model-b70f7c-kserve-7cd6d59d59-s9wps\" (UID: \"3c9174a6-1dc9-44a0-abc8-21798694cf3d\") " pod="ai-tenant-e2e-shared-a-b70f7c/auth-test-model-b70f7c-kserve-7cd6d59d59-s9wps"
Jul 29 04:46:03.447405 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:46:03.447316    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c9174a6-1dc9-44a0-abc8-21798694cf3d-model-cache\") pod \"auth-test-model-b70f7c-kserve-7cd6d59d59-s9wps\" (UID: \"3c9174a6-1dc9-44a0-abc8-21798694cf3d\") " pod="ai-tenant-e2e-shared-a-b70f7c/auth-test-model-b70f7c-kserve-7cd6d59d59-s9wps"
Jul 29 04:46:03.447405 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:46:03.447338    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c9174a6-1dc9-44a0-abc8-21798694cf3d-tls-certs\") pod \"auth-test-model-b70f7c-kserve-7cd6d59d59-s9wps\" (UID: \"3c9174a6-1dc9-44a0-abc8-21798694cf3d\") " pod="ai-tenant-e2e-shared-a-b70f7c/auth-test-model-b70f7c-kserve-7cd6d59d59-s9wps"
Jul 29 04:46:03.447405 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:46:03.447353    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c9174a6-1dc9-44a0-abc8-21798694cf3d-home\") pod \"auth-test-model-b70f7c-kserve-7cd6d59d59-s9wps\" (UID: \"3c9174a6-1dc9-44a0-abc8-21798694cf3d\") " pod="ai-tenant-e2e-shared-a-b70f7c/auth-test-model-b70f7c-kserve-7cd6d59d59-s9wps"
Jul 29 04:46:03.447405 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:46:03.447374    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c9174a6-1dc9-44a0-abc8-21798694cf3d-tmp-dir\") pod \"auth-test-model-b70f7c-kserve-7cd6d59d59-s9wps\" (UID: \"3c9174a6-1dc9-44a0-abc8-21798694cf3d\") " pod="ai-tenant-e2e-shared-a-b70f7c/auth-test-model-b70f7c-kserve-7cd6d59d59-s9wps"
Jul 29 04:46:03.548757 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:46:03.548723    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c9174a6-1dc9-44a0-abc8-21798694cf3d-model-cache\") pod \"auth-test-model-b70f7c-kserve-7cd6d59d59-s9wps\" (UID: \"3c9174a6-1dc9-44a0-abc8-21798694cf3d\") " pod="ai-tenant-e2e-shared-a-b70f7c/auth-test-model-b70f7c-kserve-7cd6d59d59-s9wps"
Jul 29 04:46:03.548757 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:46:03.548763    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c9174a6-1dc9-44a0-abc8-21798694cf3d-tls-certs\") pod \"auth-test-model-b70f7c-kserve-7cd6d59d59-s9wps\" (UID: \"3c9174a6-1dc9-44a0-abc8-21798694cf3d\") " pod="ai-tenant-e2e-shared-a-b70f7c/auth-test-model-b70f7c-kserve-7cd6d59d59-s9wps"
Jul 29 04:46:03.549008 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:46:03.548782    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c9174a6-1dc9-44a0-abc8-21798694cf3d-home\") pod \"auth-test-model-b70f7c-kserve-7cd6d59d59-s9wps\" (UID: \"3c9174a6-1dc9-44a0-abc8-21798694cf3d\") " pod="ai-tenant-e2e-shared-a-b70f7c/auth-test-model-b70f7c-kserve-7cd6d59d59-s9wps"
Jul 29 04:46:03.549008 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:46:03.548903    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c9174a6-1dc9-44a0-abc8-21798694cf3d-tmp-dir\") pod \"auth-test-model-b70f7c-kserve-7cd6d59d59-s9wps\" (UID: \"3c9174a6-1dc9-44a0-abc8-21798694cf3d\") " pod="ai-tenant-e2e-shared-a-b70f7c/auth-test-model-b70f7c-kserve-7cd6d59d59-s9wps"
Jul 29 04:46:03.549008 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:46:03.548937    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c9174a6-1dc9-44a0-abc8-21798694cf3d-dshm\") pod \"auth-test-model-b70f7c-kserve-7cd6d59d59-s9wps\" (UID: \"3c9174a6-1dc9-44a0-abc8-21798694cf3d\") " pod="ai-tenant-e2e-shared-a-b70f7c/auth-test-model-b70f7c-kserve-7cd6d59d59-s9wps"
Jul 29 04:46:03.549167 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:46:03.549086    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrqvq\" (UniqueName: \"kubernetes.io/projected/3c9174a6-1dc9-44a0-abc8-21798694cf3d-kube-api-access-jrqvq\") pod \"auth-test-model-b70f7c-kserve-7cd6d59d59-s9wps\" (UID: \"3c9174a6-1dc9-44a0-abc8-21798694cf3d\") " pod="ai-tenant-e2e-shared-a-b70f7c/auth-test-model-b70f7c-kserve-7cd6d59d59-s9wps"
Jul 29 04:46:03.549167 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:46:03.549146    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c9174a6-1dc9-44a0-abc8-21798694cf3d-model-cache\") pod \"auth-test-model-b70f7c-kserve-7cd6d59d59-s9wps\" (UID: \"3c9174a6-1dc9-44a0-abc8-21798694cf3d\") " pod="ai-tenant-e2e-shared-a-b70f7c/auth-test-model-b70f7c-kserve-7cd6d59d59-s9wps"
Jul 29 04:46:03.549276 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:46:03.549195    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c9174a6-1dc9-44a0-abc8-21798694cf3d-home\") pod \"auth-test-model-b70f7c-kserve-7cd6d59d59-s9wps\" (UID: \"3c9174a6-1dc9-44a0-abc8-21798694cf3d\") " pod="ai-tenant-e2e-shared-a-b70f7c/auth-test-model-b70f7c-kserve-7cd6d59d59-s9wps"
Jul 29 04:46:03.549339 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:46:03.549315    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c9174a6-1dc9-44a0-abc8-21798694cf3d-tmp-dir\") pod \"auth-test-model-b70f7c-kserve-7cd6d59d59-s9wps\" (UID: \"3c9174a6-1dc9-44a0-abc8-21798694cf3d\") " pod="ai-tenant-e2e-shared-a-b70f7c/auth-test-model-b70f7c-kserve-7cd6d59d59-s9wps"
Jul 29 04:46:03.551259 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:46:03.551237    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c9174a6-1dc9-44a0-abc8-21798694cf3d-dshm\") pod \"auth-test-model-b70f7c-kserve-7cd6d59d59-s9wps\" (UID: \"3c9174a6-1dc9-44a0-abc8-21798694cf3d\") " pod="ai-tenant-e2e-shared-a-b70f7c/auth-test-model-b70f7c-kserve-7cd6d59d59-s9wps"
Jul 29 04:46:03.551430 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:46:03.551412    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c9174a6-1dc9-44a0-abc8-21798694cf3d-tls-certs\") pod \"auth-test-model-b70f7c-kserve-7cd6d59d59-s9wps\" (UID: \"3c9174a6-1dc9-44a0-abc8-21798694cf3d\") " pod="ai-tenant-e2e-shared-a-b70f7c/auth-test-model-b70f7c-kserve-7cd6d59d59-s9wps"
Jul 29 04:46:03.566493 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:46:03.566464    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrqvq\" (UniqueName: \"kubernetes.io/projected/3c9174a6-1dc9-44a0-abc8-21798694cf3d-kube-api-access-jrqvq\") pod \"auth-test-model-b70f7c-kserve-7cd6d59d59-s9wps\" (UID: \"3c9174a6-1dc9-44a0-abc8-21798694cf3d\") " pod="ai-tenant-e2e-shared-a-b70f7c/auth-test-model-b70f7c-kserve-7cd6d59d59-s9wps"
Jul 29 04:46:03.643330 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:46:03.643302    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b70f7c/auth-test-model-b70f7c-kserve-7cd6d59d59-s9wps"
Jul 29 04:46:03.778553 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:46:03.778520    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-b70f7c/auth-test-model-b70f7c-kserve-7cd6d59d59-s9wps"]
Jul 29 04:46:03.781788 ip-10-0-135-138 kubenswrapper[2570]: W0729 04:46:03.781764    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c9174a6_1dc9_44a0_abc8_21798694cf3d.slice/crio-57db9c6785b8ca609aa63ba8632a6c3bad571d6bdb46a4983ed8a56c96fad624 WatchSource:0}: Error finding container 57db9c6785b8ca609aa63ba8632a6c3bad571d6bdb46a4983ed8a56c96fad624: Status 404 returned error can't find the container with id 57db9c6785b8ca609aa63ba8632a6c3bad571d6bdb46a4983ed8a56c96fad624
Jul 29 04:46:03.818728 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:46:03.818702    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b70f7c/auth-test-model-b70f7c-kserve-7cd6d59d59-s9wps" event={"ID":"3c9174a6-1dc9-44a0-abc8-21798694cf3d","Type":"ContainerStarted","Data":"57db9c6785b8ca609aa63ba8632a6c3bad571d6bdb46a4983ed8a56c96fad624"}
Jul 29 04:46:04.826055 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:46:04.826018    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b70f7c/auth-test-model-b70f7c-kserve-7cd6d59d59-s9wps" event={"ID":"3c9174a6-1dc9-44a0-abc8-21798694cf3d","Type":"ContainerStarted","Data":"4bf6b3a47ca9202fce68caf2a840c65ff26f9613124fc65f7e19573a43675964"}
Jul 29 04:46:04.843056 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:46:04.843011    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-b70f7c/auth-test-model-b70f7c-kserve-7cd6d59d59-s9wps" podStartSLOduration=1.8429983399999998 podStartE2EDuration="1.84299834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:46:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:46:04.842693749 +0000 UTC m=+3029.572824821" watchObservedRunningTime="2026-07-29 04:46:04.84299834 +0000 UTC m=+3029.573129422"
Jul 29 04:46:13.643659 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:46:13.643622    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-b70f7c/auth-test-model-b70f7c-kserve-7cd6d59d59-s9wps"
Jul 29 04:46:13.644152 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:46:13.643671    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-b70f7c/auth-test-model-b70f7c-kserve-7cd6d59d59-s9wps"
Jul 29 04:46:13.656035 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:46:13.656014    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-b70f7c/auth-test-model-b70f7c-kserve-7cd6d59d59-s9wps"
Jul 29 04:46:13.886986 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:46:13.886957    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-b70f7c/auth-test-model-b70f7c-kserve-7cd6d59d59-s9wps"
Jul 29 04:49:36.193721 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:49:36.193641    2570 scope.go:117] "RemoveContainer" containerID="d4602720ec15fc51563a9915b569a2c02dcf46af11e671552d3f4b8ecc3766ba"
Jul 29 04:50:36.212588 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:50:36.212559    2570 scope.go:117] "RemoveContainer" containerID="e437132bde0ec9e01e868bf5610cc537d94301f0b14b69780765de20f8759bdd"
Jul 29 04:51:29.404730 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:51:29.404646    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-b70f7c/auth-test-model-b70f7c-kserve-7cd6d59d59-s9wps"]
Jul 29 04:51:44.424733 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:51:44.424670    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-b70f7c/auth-test-model-b70f7c-kserve-7cd6d59d59-s9wps" podUID="3c9174a6-1dc9-44a0-abc8-21798694cf3d" containerName="main" containerID="cri-o://4bf6b3a47ca9202fce68caf2a840c65ff26f9613124fc65f7e19573a43675964" gracePeriod=45
Jul 29 04:51:44.580969 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:51:44.580928    2570 generic.go:356] "Generic (PLEG): container finished" podID="3c9174a6-1dc9-44a0-abc8-21798694cf3d" containerID="4bf6b3a47ca9202fce68caf2a840c65ff26f9613124fc65f7e19573a43675964" exitCode=0
Jul 29 04:51:44.580969 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:51:44.580958    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b70f7c/auth-test-model-b70f7c-kserve-7cd6d59d59-s9wps" event={"ID":"3c9174a6-1dc9-44a0-abc8-21798694cf3d","Type":"ContainerDied","Data":"4bf6b3a47ca9202fce68caf2a840c65ff26f9613124fc65f7e19573a43675964"}
Jul 29 04:51:44.683832 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:51:44.683760    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b70f7c/auth-test-model-b70f7c-kserve-7cd6d59d59-s9wps"
Jul 29 04:51:44.856622 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:51:44.856590    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c9174a6-1dc9-44a0-abc8-21798694cf3d-tls-certs\") pod \"3c9174a6-1dc9-44a0-abc8-21798694cf3d\" (UID: \"3c9174a6-1dc9-44a0-abc8-21798694cf3d\") "
Jul 29 04:51:44.856782 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:51:44.856655    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jrqvq\" (UniqueName: \"kubernetes.io/projected/3c9174a6-1dc9-44a0-abc8-21798694cf3d-kube-api-access-jrqvq\") pod \"3c9174a6-1dc9-44a0-abc8-21798694cf3d\" (UID: \"3c9174a6-1dc9-44a0-abc8-21798694cf3d\") "
Jul 29 04:51:44.856782 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:51:44.856672    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c9174a6-1dc9-44a0-abc8-21798694cf3d-model-cache\") pod \"3c9174a6-1dc9-44a0-abc8-21798694cf3d\" (UID: \"3c9174a6-1dc9-44a0-abc8-21798694cf3d\") "
Jul 29 04:51:44.856782 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:51:44.856700    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c9174a6-1dc9-44a0-abc8-21798694cf3d-dshm\") pod \"3c9174a6-1dc9-44a0-abc8-21798694cf3d\" (UID: \"3c9174a6-1dc9-44a0-abc8-21798694cf3d\") "
Jul 29 04:51:44.856782 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:51:44.856748    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c9174a6-1dc9-44a0-abc8-21798694cf3d-home\") pod \"3c9174a6-1dc9-44a0-abc8-21798694cf3d\" (UID: \"3c9174a6-1dc9-44a0-abc8-21798694cf3d\") "
Jul 29 04:51:44.857031 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:51:44.856847    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c9174a6-1dc9-44a0-abc8-21798694cf3d-tmp-dir\") pod \"3c9174a6-1dc9-44a0-abc8-21798694cf3d\" (UID: \"3c9174a6-1dc9-44a0-abc8-21798694cf3d\") "
Jul 29 04:51:44.857031 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:51:44.857007    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c9174a6-1dc9-44a0-abc8-21798694cf3d-model-cache" (OuterVolumeSpecName: "model-cache") pod "3c9174a6-1dc9-44a0-abc8-21798694cf3d" (UID: "3c9174a6-1dc9-44a0-abc8-21798694cf3d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 04:51:44.857145 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:51:44.857097    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c9174a6-1dc9-44a0-abc8-21798694cf3d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3c9174a6-1dc9-44a0-abc8-21798694cf3d" (UID: "3c9174a6-1dc9-44a0-abc8-21798694cf3d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 04:51:44.857145 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:51:44.857110    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c9174a6-1dc9-44a0-abc8-21798694cf3d-home" (OuterVolumeSpecName: "home") pod "3c9174a6-1dc9-44a0-abc8-21798694cf3d" (UID: "3c9174a6-1dc9-44a0-abc8-21798694cf3d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 04:51:44.857323 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:51:44.857305    2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c9174a6-1dc9-44a0-abc8-21798694cf3d-model-cache\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:51:44.857385 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:51:44.857327    2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c9174a6-1dc9-44a0-abc8-21798694cf3d-home\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:51:44.857385 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:51:44.857340    2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c9174a6-1dc9-44a0-abc8-21798694cf3d-tmp-dir\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:51:44.859143 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:51:44.859104    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c9174a6-1dc9-44a0-abc8-21798694cf3d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3c9174a6-1dc9-44a0-abc8-21798694cf3d" (UID: "3c9174a6-1dc9-44a0-abc8-21798694cf3d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 04:51:44.859143 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:51:44.859137    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c9174a6-1dc9-44a0-abc8-21798694cf3d-dshm" (OuterVolumeSpecName: "dshm") pod "3c9174a6-1dc9-44a0-abc8-21798694cf3d" (UID: "3c9174a6-1dc9-44a0-abc8-21798694cf3d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 04:51:44.859260 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:51:44.859200    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c9174a6-1dc9-44a0-abc8-21798694cf3d-kube-api-access-jrqvq" (OuterVolumeSpecName: "kube-api-access-jrqvq") pod "3c9174a6-1dc9-44a0-abc8-21798694cf3d" (UID: "3c9174a6-1dc9-44a0-abc8-21798694cf3d"). InnerVolumeSpecName "kube-api-access-jrqvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 04:51:44.957922 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:51:44.957847    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c9174a6-1dc9-44a0-abc8-21798694cf3d-tls-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:51:44.957922 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:51:44.957873    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrqvq\" (UniqueName: \"kubernetes.io/projected/3c9174a6-1dc9-44a0-abc8-21798694cf3d-kube-api-access-jrqvq\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:51:44.957922 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:51:44.957884    2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c9174a6-1dc9-44a0-abc8-21798694cf3d-dshm\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:51:45.587255 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:51:45.587224    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b70f7c/auth-test-model-b70f7c-kserve-7cd6d59d59-s9wps"
Jul 29 04:51:45.587736 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:51:45.587267    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b70f7c/auth-test-model-b70f7c-kserve-7cd6d59d59-s9wps" event={"ID":"3c9174a6-1dc9-44a0-abc8-21798694cf3d","Type":"ContainerDied","Data":"57db9c6785b8ca609aa63ba8632a6c3bad571d6bdb46a4983ed8a56c96fad624"}
Jul 29 04:51:45.587736 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:51:45.587312    2570 scope.go:117] "RemoveContainer" containerID="4bf6b3a47ca9202fce68caf2a840c65ff26f9613124fc65f7e19573a43675964"
Jul 29 04:51:45.613786 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:51:45.611012    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-b70f7c/auth-test-model-b70f7c-kserve-7cd6d59d59-s9wps"]
Jul 29 04:51:45.617282 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:51:45.617249    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-b70f7c/auth-test-model-b70f7c-kserve-7cd6d59d59-s9wps"]
Jul 29 04:51:45.835415 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:51:45.835380    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c9174a6-1dc9-44a0-abc8-21798694cf3d" path="/var/lib/kubelet/pods/3c9174a6-1dc9-44a0-abc8-21798694cf3d/volumes"
Jul 29 04:52:43.929192 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:52:43.929157    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-b550f6/sub-test-model-b550f6-kserve-5f4f8cc44c-65j96"]
Jul 29 04:52:43.929605 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:52:43.929591    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c9174a6-1dc9-44a0-abc8-21798694cf3d" containerName="main"
Jul 29 04:52:43.929653 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:52:43.929607    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c9174a6-1dc9-44a0-abc8-21798694cf3d" containerName="main"
Jul 29 04:52:43.929689 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:52:43.929669    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c9174a6-1dc9-44a0-abc8-21798694cf3d" containerName="main"
Jul 29 04:52:43.934434 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:52:43.934414    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b550f6/sub-test-model-b550f6-kserve-5f4f8cc44c-65j96"
Jul 29 04:52:43.941086 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:52:43.938499    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b550f6\"/\"openshift-service-ca.crt\""
Jul 29 04:52:43.941086 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:52:43.938669    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b550f6\"/\"default-dockercfg-95cjw\""
Jul 29 04:52:43.941086 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:52:43.938900    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b550f6\"/\"sub-test-model-b550f6-kserve-self-signed-certs\""
Jul 29 04:52:43.941086 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:52:43.938952    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b550f6\"/\"kube-root-ca.crt\""
Jul 29 04:52:43.951146 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:52:43.951120    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b550f6/sub-test-model-b550f6-kserve-5f4f8cc44c-65j96"]
Jul 29 04:52:43.979397 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:52:43.979368    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e8f64fc-3471-4636-a5a3-dcdbb65b6a73-dshm\") pod \"sub-test-model-b550f6-kserve-5f4f8cc44c-65j96\" (UID: \"0e8f64fc-3471-4636-a5a3-dcdbb65b6a73\") " pod="ai-tenant-e2e-shared-b-b550f6/sub-test-model-b550f6-kserve-5f4f8cc44c-65j96"
Jul 29 04:52:43.979527 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:52:43.979431    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e8f64fc-3471-4636-a5a3-dcdbb65b6a73-tmp-dir\") pod \"sub-test-model-b550f6-kserve-5f4f8cc44c-65j96\" (UID: \"0e8f64fc-3471-4636-a5a3-dcdbb65b6a73\") " pod="ai-tenant-e2e-shared-b-b550f6/sub-test-model-b550f6-kserve-5f4f8cc44c-65j96"
Jul 29 04:52:43.979527 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:52:43.979469    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e8f64fc-3471-4636-a5a3-dcdbb65b6a73-home\") pod \"sub-test-model-b550f6-kserve-5f4f8cc44c-65j96\" (UID: \"0e8f64fc-3471-4636-a5a3-dcdbb65b6a73\") " pod="ai-tenant-e2e-shared-b-b550f6/sub-test-model-b550f6-kserve-5f4f8cc44c-65j96"
Jul 29 04:52:43.979527 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:52:43.979502    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxzxr\" (UniqueName: \"kubernetes.io/projected/0e8f64fc-3471-4636-a5a3-dcdbb65b6a73-kube-api-access-rxzxr\") pod \"sub-test-model-b550f6-kserve-5f4f8cc44c-65j96\" (UID: \"0e8f64fc-3471-4636-a5a3-dcdbb65b6a73\") " pod="ai-tenant-e2e-shared-b-b550f6/sub-test-model-b550f6-kserve-5f4f8cc44c-65j96"
Jul 29 04:52:43.979641 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:52:43.979576    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e8f64fc-3471-4636-a5a3-dcdbb65b6a73-tls-certs\") pod \"sub-test-model-b550f6-kserve-5f4f8cc44c-65j96\" (UID: \"0e8f64fc-3471-4636-a5a3-dcdbb65b6a73\") " pod="ai-tenant-e2e-shared-b-b550f6/sub-test-model-b550f6-kserve-5f4f8cc44c-65j96"
Jul 29 04:52:43.979641 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:52:43.979626    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e8f64fc-3471-4636-a5a3-dcdbb65b6a73-model-cache\") pod \"sub-test-model-b550f6-kserve-5f4f8cc44c-65j96\" (UID: \"0e8f64fc-3471-4636-a5a3-dcdbb65b6a73\") " pod="ai-tenant-e2e-shared-b-b550f6/sub-test-model-b550f6-kserve-5f4f8cc44c-65j96"
Jul 29 04:52:44.080126 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:52:44.080098    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e8f64fc-3471-4636-a5a3-dcdbb65b6a73-home\") pod \"sub-test-model-b550f6-kserve-5f4f8cc44c-65j96\" (UID: \"0e8f64fc-3471-4636-a5a3-dcdbb65b6a73\") " pod="ai-tenant-e2e-shared-b-b550f6/sub-test-model-b550f6-kserve-5f4f8cc44c-65j96"
Jul 29 04:52:44.080294 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:52:44.080132    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxzxr\" (UniqueName: \"kubernetes.io/projected/0e8f64fc-3471-4636-a5a3-dcdbb65b6a73-kube-api-access-rxzxr\") pod \"sub-test-model-b550f6-kserve-5f4f8cc44c-65j96\" (UID: \"0e8f64fc-3471-4636-a5a3-dcdbb65b6a73\") " pod="ai-tenant-e2e-shared-b-b550f6/sub-test-model-b550f6-kserve-5f4f8cc44c-65j96"
Jul 29 04:52:44.080294 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:52:44.080162    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e8f64fc-3471-4636-a5a3-dcdbb65b6a73-tls-certs\") pod \"sub-test-model-b550f6-kserve-5f4f8cc44c-65j96\" (UID: \"0e8f64fc-3471-4636-a5a3-dcdbb65b6a73\") " pod="ai-tenant-e2e-shared-b-b550f6/sub-test-model-b550f6-kserve-5f4f8cc44c-65j96"
Jul 29 04:52:44.080294 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:52:44.080201    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e8f64fc-3471-4636-a5a3-dcdbb65b6a73-model-cache\") pod \"sub-test-model-b550f6-kserve-5f4f8cc44c-65j96\" (UID: \"0e8f64fc-3471-4636-a5a3-dcdbb65b6a73\") " pod="ai-tenant-e2e-shared-b-b550f6/sub-test-model-b550f6-kserve-5f4f8cc44c-65j96"
Jul 29 04:52:44.080294 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:52:44.080260    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e8f64fc-3471-4636-a5a3-dcdbb65b6a73-dshm\") pod \"sub-test-model-b550f6-kserve-5f4f8cc44c-65j96\" (UID: \"0e8f64fc-3471-4636-a5a3-dcdbb65b6a73\") " pod="ai-tenant-e2e-shared-b-b550f6/sub-test-model-b550f6-kserve-5f4f8cc44c-65j96"
Jul 29 04:52:44.080499 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:52:44.080299    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e8f64fc-3471-4636-a5a3-dcdbb65b6a73-tmp-dir\") pod \"sub-test-model-b550f6-kserve-5f4f8cc44c-65j96\" (UID: \"0e8f64fc-3471-4636-a5a3-dcdbb65b6a73\") " pod="ai-tenant-e2e-shared-b-b550f6/sub-test-model-b550f6-kserve-5f4f8cc44c-65j96"
Jul 29 04:52:44.080550 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:52:44.080527    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e8f64fc-3471-4636-a5a3-dcdbb65b6a73-home\") pod \"sub-test-model-b550f6-kserve-5f4f8cc44c-65j96\" (UID: \"0e8f64fc-3471-4636-a5a3-dcdbb65b6a73\") " pod="ai-tenant-e2e-shared-b-b550f6/sub-test-model-b550f6-kserve-5f4f8cc44c-65j96"
Jul 29 04:52:44.080550 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:52:44.080540    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e8f64fc-3471-4636-a5a3-dcdbb65b6a73-model-cache\") pod \"sub-test-model-b550f6-kserve-5f4f8cc44c-65j96\" (UID: \"0e8f64fc-3471-4636-a5a3-dcdbb65b6a73\") " pod="ai-tenant-e2e-shared-b-b550f6/sub-test-model-b550f6-kserve-5f4f8cc44c-65j96"
Jul 29 04:52:44.080614 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:52:44.080589    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e8f64fc-3471-4636-a5a3-dcdbb65b6a73-tmp-dir\") pod \"sub-test-model-b550f6-kserve-5f4f8cc44c-65j96\" (UID: \"0e8f64fc-3471-4636-a5a3-dcdbb65b6a73\") " pod="ai-tenant-e2e-shared-b-b550f6/sub-test-model-b550f6-kserve-5f4f8cc44c-65j96"
Jul 29 04:52:44.082738 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:52:44.082719    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e8f64fc-3471-4636-a5a3-dcdbb65b6a73-dshm\") pod \"sub-test-model-b550f6-kserve-5f4f8cc44c-65j96\" (UID: \"0e8f64fc-3471-4636-a5a3-dcdbb65b6a73\") " pod="ai-tenant-e2e-shared-b-b550f6/sub-test-model-b550f6-kserve-5f4f8cc44c-65j96"
Jul 29 04:52:44.082811 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:52:44.082773    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e8f64fc-3471-4636-a5a3-dcdbb65b6a73-tls-certs\") pod \"sub-test-model-b550f6-kserve-5f4f8cc44c-65j96\" (UID: \"0e8f64fc-3471-4636-a5a3-dcdbb65b6a73\") " pod="ai-tenant-e2e-shared-b-b550f6/sub-test-model-b550f6-kserve-5f4f8cc44c-65j96"
Jul 29 04:52:44.087299 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:52:44.087279    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxzxr\" (UniqueName: \"kubernetes.io/projected/0e8f64fc-3471-4636-a5a3-dcdbb65b6a73-kube-api-access-rxzxr\") pod \"sub-test-model-b550f6-kserve-5f4f8cc44c-65j96\" (UID: \"0e8f64fc-3471-4636-a5a3-dcdbb65b6a73\") " pod="ai-tenant-e2e-shared-b-b550f6/sub-test-model-b550f6-kserve-5f4f8cc44c-65j96"
Jul 29 04:52:44.252664 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:52:44.252579    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b550f6/sub-test-model-b550f6-kserve-5f4f8cc44c-65j96"
Jul 29 04:52:44.386918 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:52:44.386896    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b550f6/sub-test-model-b550f6-kserve-5f4f8cc44c-65j96"]
Jul 29 04:52:44.388797 ip-10-0-135-138 kubenswrapper[2570]: W0729 04:52:44.388772    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0e8f64fc_3471_4636_a5a3_dcdbb65b6a73.slice/crio-5c0ce3a4307a4b1aa252348b927aa03a759b55a1a06be66b7eae767e36e6c0c7 WatchSource:0}: Error finding container 5c0ce3a4307a4b1aa252348b927aa03a759b55a1a06be66b7eae767e36e6c0c7: Status 404 returned error can't find the container with id 5c0ce3a4307a4b1aa252348b927aa03a759b55a1a06be66b7eae767e36e6c0c7
Jul 29 04:52:44.880030 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:52:44.879992    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b550f6/sub-test-model-b550f6-kserve-5f4f8cc44c-65j96" event={"ID":"0e8f64fc-3471-4636-a5a3-dcdbb65b6a73","Type":"ContainerStarted","Data":"1577e6600cbf4c8316f2297a673413e1abf3f2fe900d104a3f101d0117c621fc"}
Jul 29 04:52:44.880030 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:52:44.880035    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b550f6/sub-test-model-b550f6-kserve-5f4f8cc44c-65j96" event={"ID":"0e8f64fc-3471-4636-a5a3-dcdbb65b6a73","Type":"ContainerStarted","Data":"5c0ce3a4307a4b1aa252348b927aa03a759b55a1a06be66b7eae767e36e6c0c7"}
Jul 29 04:52:44.895585 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:52:44.895535    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-b550f6/sub-test-model-b550f6-kserve-5f4f8cc44c-65j96" podStartSLOduration=1.895513322 podStartE2EDuration="1.895513322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:52:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:52:44.895302653 +0000 UTC m=+3429.625433724" watchObservedRunningTime="2026-07-29 04:52:44.895513322 +0000 UTC m=+3429.625644392"
Jul 29 04:52:54.253129 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:52:54.253087    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-b550f6/sub-test-model-b550f6-kserve-5f4f8cc44c-65j96"
Jul 29 04:52:54.253590 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:52:54.253310    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-b550f6/sub-test-model-b550f6-kserve-5f4f8cc44c-65j96"
Jul 29 04:52:54.265756 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:52:54.265731    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-b550f6/sub-test-model-b550f6-kserve-5f4f8cc44c-65j96"
Jul 29 04:52:54.945558 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:52:54.945527    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-b550f6/sub-test-model-b550f6-kserve-5f4f8cc44c-65j96"
Jul 29 04:54:27.021253 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:54:27.021169    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b550f6/sub-test-model-b550f6-kserve-5f4f8cc44c-65j96"]
Jul 29 04:54:34.335470 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:54:34.335439    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-b70f7c/rate-test-model-b70f7c-kserve-7c4d9df799-qr8p2"]
Jul 29 04:54:34.339106 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:54:34.339091    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b70f7c/rate-test-model-b70f7c-kserve-7c4d9df799-qr8p2"
Jul 29 04:54:34.342763 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:54:34.342732    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-b70f7c\"/\"default-dockercfg-4rfcc\""
Jul 29 04:54:34.342918 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:54:34.342779    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-b70f7c\"/\"kube-root-ca.crt\""
Jul 29 04:54:34.342918 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:54:34.342732    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-b70f7c\"/\"openshift-service-ca.crt\""
Jul 29 04:54:34.342918 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:54:34.342788    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-b70f7c\"/\"rate-test-model-b70f7c-kserve-self-signed-certs\""
Jul 29 04:54:34.348132 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:54:34.348109    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-b70f7c/rate-test-model-b70f7c-kserve-7c4d9df799-qr8p2"]
Jul 29 04:54:34.509274 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:54:34.509240    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75a76190-e96a-46bb-bd22-da051039ebb3-tls-certs\") pod \"rate-test-model-b70f7c-kserve-7c4d9df799-qr8p2\" (UID: \"75a76190-e96a-46bb-bd22-da051039ebb3\") " pod="ai-tenant-e2e-shared-a-b70f7c/rate-test-model-b70f7c-kserve-7c4d9df799-qr8p2"
Jul 29 04:54:34.509440 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:54:34.509310    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75a76190-e96a-46bb-bd22-da051039ebb3-dshm\") pod \"rate-test-model-b70f7c-kserve-7c4d9df799-qr8p2\" (UID: \"75a76190-e96a-46bb-bd22-da051039ebb3\") " pod="ai-tenant-e2e-shared-a-b70f7c/rate-test-model-b70f7c-kserve-7c4d9df799-qr8p2"
Jul 29 04:54:34.509440 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:54:34.509366    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75a76190-e96a-46bb-bd22-da051039ebb3-tmp-dir\") pod \"rate-test-model-b70f7c-kserve-7c4d9df799-qr8p2\" (UID: \"75a76190-e96a-46bb-bd22-da051039ebb3\") " pod="ai-tenant-e2e-shared-a-b70f7c/rate-test-model-b70f7c-kserve-7c4d9df799-qr8p2"
Jul 29 04:54:34.509440 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:54:34.509386    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75a76190-e96a-46bb-bd22-da051039ebb3-home\") pod \"rate-test-model-b70f7c-kserve-7c4d9df799-qr8p2\" (UID: \"75a76190-e96a-46bb-bd22-da051039ebb3\") " pod="ai-tenant-e2e-shared-a-b70f7c/rate-test-model-b70f7c-kserve-7c4d9df799-qr8p2"
Jul 29 04:54:34.509440 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:54:34.509413    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75a76190-e96a-46bb-bd22-da051039ebb3-model-cache\") pod \"rate-test-model-b70f7c-kserve-7c4d9df799-qr8p2\" (UID: \"75a76190-e96a-46bb-bd22-da051039ebb3\") " pod="ai-tenant-e2e-shared-a-b70f7c/rate-test-model-b70f7c-kserve-7c4d9df799-qr8p2"
Jul 29 04:54:34.509440 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:54:34.509430    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nwtx\" (UniqueName: \"kubernetes.io/projected/75a76190-e96a-46bb-bd22-da051039ebb3-kube-api-access-8nwtx\") pod \"rate-test-model-b70f7c-kserve-7c4d9df799-qr8p2\" (UID: \"75a76190-e96a-46bb-bd22-da051039ebb3\") " pod="ai-tenant-e2e-shared-a-b70f7c/rate-test-model-b70f7c-kserve-7c4d9df799-qr8p2"
Jul 29 04:54:34.610258 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:54:34.610188    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75a76190-e96a-46bb-bd22-da051039ebb3-tmp-dir\") pod \"rate-test-model-b70f7c-kserve-7c4d9df799-qr8p2\" (UID: \"75a76190-e96a-46bb-bd22-da051039ebb3\") " pod="ai-tenant-e2e-shared-a-b70f7c/rate-test-model-b70f7c-kserve-7c4d9df799-qr8p2"
Jul 29 04:54:34.610258 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:54:34.610219    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75a76190-e96a-46bb-bd22-da051039ebb3-home\") pod \"rate-test-model-b70f7c-kserve-7c4d9df799-qr8p2\" (UID: \"75a76190-e96a-46bb-bd22-da051039ebb3\") " pod="ai-tenant-e2e-shared-a-b70f7c/rate-test-model-b70f7c-kserve-7c4d9df799-qr8p2"
Jul 29 04:54:34.610258 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:54:34.610249    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75a76190-e96a-46bb-bd22-da051039ebb3-model-cache\") pod \"rate-test-model-b70f7c-kserve-7c4d9df799-qr8p2\" (UID: \"75a76190-e96a-46bb-bd22-da051039ebb3\") " pod="ai-tenant-e2e-shared-a-b70f7c/rate-test-model-b70f7c-kserve-7c4d9df799-qr8p2"
Jul 29 04:54:34.610513 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:54:34.610273    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nwtx\" (UniqueName: \"kubernetes.io/projected/75a76190-e96a-46bb-bd22-da051039ebb3-kube-api-access-8nwtx\") pod \"rate-test-model-b70f7c-kserve-7c4d9df799-qr8p2\" (UID: \"75a76190-e96a-46bb-bd22-da051039ebb3\") " pod="ai-tenant-e2e-shared-a-b70f7c/rate-test-model-b70f7c-kserve-7c4d9df799-qr8p2"
Jul 29 04:54:34.610513 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:54:34.610315    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75a76190-e96a-46bb-bd22-da051039ebb3-tls-certs\") pod \"rate-test-model-b70f7c-kserve-7c4d9df799-qr8p2\" (UID: \"75a76190-e96a-46bb-bd22-da051039ebb3\") " pod="ai-tenant-e2e-shared-a-b70f7c/rate-test-model-b70f7c-kserve-7c4d9df799-qr8p2"
Jul 29 04:54:34.610513 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:54:34.610388    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75a76190-e96a-46bb-bd22-da051039ebb3-dshm\") pod \"rate-test-model-b70f7c-kserve-7c4d9df799-qr8p2\" (UID: \"75a76190-e96a-46bb-bd22-da051039ebb3\") " pod="ai-tenant-e2e-shared-a-b70f7c/rate-test-model-b70f7c-kserve-7c4d9df799-qr8p2"
Jul 29 04:54:34.610679 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:54:34.610630    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75a76190-e96a-46bb-bd22-da051039ebb3-model-cache\") pod \"rate-test-model-b70f7c-kserve-7c4d9df799-qr8p2\" (UID: \"75a76190-e96a-46bb-bd22-da051039ebb3\") " pod="ai-tenant-e2e-shared-a-b70f7c/rate-test-model-b70f7c-kserve-7c4d9df799-qr8p2"
Jul 29 04:54:34.610679 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:54:34.610664    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75a76190-e96a-46bb-bd22-da051039ebb3-tmp-dir\") pod \"rate-test-model-b70f7c-kserve-7c4d9df799-qr8p2\" (UID: \"75a76190-e96a-46bb-bd22-da051039ebb3\") " pod="ai-tenant-e2e-shared-a-b70f7c/rate-test-model-b70f7c-kserve-7c4d9df799-qr8p2"
Jul 29 04:54:34.610789 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:54:34.610715    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75a76190-e96a-46bb-bd22-da051039ebb3-home\") pod \"rate-test-model-b70f7c-kserve-7c4d9df799-qr8p2\" (UID: \"75a76190-e96a-46bb-bd22-da051039ebb3\") " pod="ai-tenant-e2e-shared-a-b70f7c/rate-test-model-b70f7c-kserve-7c4d9df799-qr8p2"
Jul 29 04:54:34.612894 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:54:34.612870    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75a76190-e96a-46bb-bd22-da051039ebb3-dshm\") pod \"rate-test-model-b70f7c-kserve-7c4d9df799-qr8p2\" (UID: \"75a76190-e96a-46bb-bd22-da051039ebb3\") " pod="ai-tenant-e2e-shared-a-b70f7c/rate-test-model-b70f7c-kserve-7c4d9df799-qr8p2"
Jul 29 04:54:34.613086 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:54:34.613071    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75a76190-e96a-46bb-bd22-da051039ebb3-tls-certs\") pod \"rate-test-model-b70f7c-kserve-7c4d9df799-qr8p2\" (UID: \"75a76190-e96a-46bb-bd22-da051039ebb3\") " pod="ai-tenant-e2e-shared-a-b70f7c/rate-test-model-b70f7c-kserve-7c4d9df799-qr8p2"
Jul 29 04:54:34.618613 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:54:34.618588    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nwtx\" (UniqueName: \"kubernetes.io/projected/75a76190-e96a-46bb-bd22-da051039ebb3-kube-api-access-8nwtx\") pod \"rate-test-model-b70f7c-kserve-7c4d9df799-qr8p2\" (UID: \"75a76190-e96a-46bb-bd22-da051039ebb3\") " pod="ai-tenant-e2e-shared-a-b70f7c/rate-test-model-b70f7c-kserve-7c4d9df799-qr8p2"
Jul 29 04:54:34.649479 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:54:34.649456    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b70f7c/rate-test-model-b70f7c-kserve-7c4d9df799-qr8p2"
Jul 29 04:54:34.783670 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:54:34.783639    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-b70f7c/rate-test-model-b70f7c-kserve-7c4d9df799-qr8p2"]
Jul 29 04:54:34.785087 ip-10-0-135-138 kubenswrapper[2570]: W0729 04:54:34.785064    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod75a76190_e96a_46bb_bd22_da051039ebb3.slice/crio-46a3db83f8754e99b343fd320b159288d26eb8646ef72e3202a132d4197cbef6 WatchSource:0}: Error finding container 46a3db83f8754e99b343fd320b159288d26eb8646ef72e3202a132d4197cbef6: Status 404 returned error can't find the container with id 46a3db83f8754e99b343fd320b159288d26eb8646ef72e3202a132d4197cbef6
Jul 29 04:54:35.432994 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:54:35.432959    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b70f7c/rate-test-model-b70f7c-kserve-7c4d9df799-qr8p2" event={"ID":"75a76190-e96a-46bb-bd22-da051039ebb3","Type":"ContainerStarted","Data":"e50efb917af604095e1ef4491887baa07e3b45bda40884116123ae4502218749"}
Jul 29 04:54:35.432994 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:54:35.432998    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b70f7c/rate-test-model-b70f7c-kserve-7c4d9df799-qr8p2" event={"ID":"75a76190-e96a-46bb-bd22-da051039ebb3","Type":"ContainerStarted","Data":"46a3db83f8754e99b343fd320b159288d26eb8646ef72e3202a132d4197cbef6"}
Jul 29 04:54:35.449773 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:54:35.449727    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-b70f7c/rate-test-model-b70f7c-kserve-7c4d9df799-qr8p2" podStartSLOduration=1.449713108 podStartE2EDuration="1.449713108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:54:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:54:35.448849082 +0000 UTC m=+3540.178980179" watchObservedRunningTime="2026-07-29 04:54:35.449713108 +0000 UTC m=+3540.179844177"
Jul 29 04:54:42.041154 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:54:42.041092    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-b550f6/sub-test-model-b550f6-kserve-5f4f8cc44c-65j96" podUID="0e8f64fc-3471-4636-a5a3-dcdbb65b6a73" containerName="main" containerID="cri-o://1577e6600cbf4c8316f2297a673413e1abf3f2fe900d104a3f101d0117c621fc" gracePeriod=45
Jul 29 04:54:42.301173 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:54:42.301114    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b550f6/sub-test-model-b550f6-kserve-5f4f8cc44c-65j96"
Jul 29 04:54:42.473700 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:54:42.473662    2570 generic.go:356] "Generic (PLEG): container finished" podID="0e8f64fc-3471-4636-a5a3-dcdbb65b6a73" containerID="1577e6600cbf4c8316f2297a673413e1abf3f2fe900d104a3f101d0117c621fc" exitCode=0
Jul 29 04:54:42.474049 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:54:42.474023    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b550f6/sub-test-model-b550f6-kserve-5f4f8cc44c-65j96" event={"ID":"0e8f64fc-3471-4636-a5a3-dcdbb65b6a73","Type":"ContainerDied","Data":"1577e6600cbf4c8316f2297a673413e1abf3f2fe900d104a3f101d0117c621fc"}
Jul 29 04:54:42.474197 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:54:42.474182    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b550f6/sub-test-model-b550f6-kserve-5f4f8cc44c-65j96" event={"ID":"0e8f64fc-3471-4636-a5a3-dcdbb65b6a73","Type":"ContainerDied","Data":"5c0ce3a4307a4b1aa252348b927aa03a759b55a1a06be66b7eae767e36e6c0c7"}
Jul 29 04:54:42.474299 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:54:42.474287    2570 scope.go:117] "RemoveContainer" containerID="1577e6600cbf4c8316f2297a673413e1abf3f2fe900d104a3f101d0117c621fc"
Jul 29 04:54:42.474564 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:54:42.474547    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b550f6/sub-test-model-b550f6-kserve-5f4f8cc44c-65j96"
Jul 29 04:54:42.484840 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:54:42.484798    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rxzxr\" (UniqueName: \"kubernetes.io/projected/0e8f64fc-3471-4636-a5a3-dcdbb65b6a73-kube-api-access-rxzxr\") pod \"0e8f64fc-3471-4636-a5a3-dcdbb65b6a73\" (UID: \"0e8f64fc-3471-4636-a5a3-dcdbb65b6a73\") "
Jul 29 04:54:42.484953 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:54:42.484858    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e8f64fc-3471-4636-a5a3-dcdbb65b6a73-model-cache\") pod \"0e8f64fc-3471-4636-a5a3-dcdbb65b6a73\" (UID: \"0e8f64fc-3471-4636-a5a3-dcdbb65b6a73\") "
Jul 29 04:54:42.484953 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:54:42.484935    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e8f64fc-3471-4636-a5a3-dcdbb65b6a73-tmp-dir\") pod \"0e8f64fc-3471-4636-a5a3-dcdbb65b6a73\" (UID: \"0e8f64fc-3471-4636-a5a3-dcdbb65b6a73\") "
Jul 29 04:54:42.485068 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:54:42.484977    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e8f64fc-3471-4636-a5a3-dcdbb65b6a73-dshm\") pod \"0e8f64fc-3471-4636-a5a3-dcdbb65b6a73\" (UID: \"0e8f64fc-3471-4636-a5a3-dcdbb65b6a73\") "
Jul 29 04:54:42.485068 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:54:42.485046    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e8f64fc-3471-4636-a5a3-dcdbb65b6a73-home\") pod \"0e8f64fc-3471-4636-a5a3-dcdbb65b6a73\" (UID: \"0e8f64fc-3471-4636-a5a3-dcdbb65b6a73\") "
Jul 29 04:54:42.485178 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:54:42.485070    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e8f64fc-3471-4636-a5a3-dcdbb65b6a73-tls-certs\") pod \"0e8f64fc-3471-4636-a5a3-dcdbb65b6a73\" (UID: \"0e8f64fc-3471-4636-a5a3-dcdbb65b6a73\") "
Jul 29 04:54:42.485235 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:54:42.485201    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e8f64fc-3471-4636-a5a3-dcdbb65b6a73-model-cache" (OuterVolumeSpecName: "model-cache") pod "0e8f64fc-3471-4636-a5a3-dcdbb65b6a73" (UID: "0e8f64fc-3471-4636-a5a3-dcdbb65b6a73"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 04:54:42.485458 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:54:42.485437    2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e8f64fc-3471-4636-a5a3-dcdbb65b6a73-model-cache\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:54:42.485745 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:54:42.485665    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e8f64fc-3471-4636-a5a3-dcdbb65b6a73-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0e8f64fc-3471-4636-a5a3-dcdbb65b6a73" (UID: "0e8f64fc-3471-4636-a5a3-dcdbb65b6a73"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 04:54:42.485889 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:54:42.485871    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e8f64fc-3471-4636-a5a3-dcdbb65b6a73-home" (OuterVolumeSpecName: "home") pod "0e8f64fc-3471-4636-a5a3-dcdbb65b6a73" (UID: "0e8f64fc-3471-4636-a5a3-dcdbb65b6a73"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 04:54:42.487748 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:54:42.487721    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e8f64fc-3471-4636-a5a3-dcdbb65b6a73-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0e8f64fc-3471-4636-a5a3-dcdbb65b6a73" (UID: "0e8f64fc-3471-4636-a5a3-dcdbb65b6a73"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 04:54:42.488004 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:54:42.487973    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e8f64fc-3471-4636-a5a3-dcdbb65b6a73-kube-api-access-rxzxr" (OuterVolumeSpecName: "kube-api-access-rxzxr") pod "0e8f64fc-3471-4636-a5a3-dcdbb65b6a73" (UID: "0e8f64fc-3471-4636-a5a3-dcdbb65b6a73"). InnerVolumeSpecName "kube-api-access-rxzxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 04:54:42.488138 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:54:42.488121    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e8f64fc-3471-4636-a5a3-dcdbb65b6a73-dshm" (OuterVolumeSpecName: "dshm") pod "0e8f64fc-3471-4636-a5a3-dcdbb65b6a73" (UID: "0e8f64fc-3471-4636-a5a3-dcdbb65b6a73"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 04:54:42.491733 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:54:42.491709    2570 scope.go:117] "RemoveContainer" containerID="1577e6600cbf4c8316f2297a673413e1abf3f2fe900d104a3f101d0117c621fc"
Jul 29 04:54:42.492048 ip-10-0-135-138 kubenswrapper[2570]: E0729 04:54:42.492027    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1577e6600cbf4c8316f2297a673413e1abf3f2fe900d104a3f101d0117c621fc\": container with ID starting with 1577e6600cbf4c8316f2297a673413e1abf3f2fe900d104a3f101d0117c621fc not found: ID does not exist" containerID="1577e6600cbf4c8316f2297a673413e1abf3f2fe900d104a3f101d0117c621fc"
Jul 29 04:54:42.492146 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:54:42.492055    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1577e6600cbf4c8316f2297a673413e1abf3f2fe900d104a3f101d0117c621fc"} err="failed to get container status \"1577e6600cbf4c8316f2297a673413e1abf3f2fe900d104a3f101d0117c621fc\": rpc error: code = NotFound desc = could not find container \"1577e6600cbf4c8316f2297a673413e1abf3f2fe900d104a3f101d0117c621fc\": container with ID starting with 1577e6600cbf4c8316f2297a673413e1abf3f2fe900d104a3f101d0117c621fc not found: ID does not exist"
Jul 29 04:54:42.586243 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:54:42.586217    2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e8f64fc-3471-4636-a5a3-dcdbb65b6a73-tmp-dir\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:54:42.586243 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:54:42.586239    2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e8f64fc-3471-4636-a5a3-dcdbb65b6a73-dshm\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:54:42.586243 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:54:42.586247    2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e8f64fc-3471-4636-a5a3-dcdbb65b6a73-home\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:54:42.586461 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:54:42.586257    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e8f64fc-3471-4636-a5a3-dcdbb65b6a73-tls-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:54:42.586461 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:54:42.586265    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxzxr\" (UniqueName: \"kubernetes.io/projected/0e8f64fc-3471-4636-a5a3-dcdbb65b6a73-kube-api-access-rxzxr\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:54:42.797081 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:54:42.797050    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b550f6/sub-test-model-b550f6-kserve-5f4f8cc44c-65j96"]
Jul 29 04:54:42.801912 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:54:42.801879    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-b550f6/sub-test-model-b550f6-kserve-5f4f8cc44c-65j96"]
Jul 29 04:54:43.835048 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:54:43.835007    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e8f64fc-3471-4636-a5a3-dcdbb65b6a73" path="/var/lib/kubelet/pods/0e8f64fc-3471-4636-a5a3-dcdbb65b6a73/volumes"
Jul 29 04:54:44.650141 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:54:44.650101    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-b70f7c/rate-test-model-b70f7c-kserve-7c4d9df799-qr8p2"
Jul 29 04:54:44.650323 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:54:44.650162    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-b70f7c/rate-test-model-b70f7c-kserve-7c4d9df799-qr8p2"
Jul 29 04:54:44.662453 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:54:44.662429    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-b70f7c/rate-test-model-b70f7c-kserve-7c4d9df799-qr8p2"
Jul 29 04:54:45.500423 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:54:45.500396    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-b70f7c/rate-test-model-b70f7c-kserve-7c4d9df799-qr8p2"
Jul 29 04:56:36.146508 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:36.146475    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-b70f7c/rate-test-model-b70f7c-kserve-7c4d9df799-qr8p2"]
Jul 29 04:56:50.489296 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:50.489259    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7677-140502348540064"]
Jul 29 04:56:50.489708 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:50.489677    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e8f64fc-3471-4636-a5a3-dcdbb65b6a73" containerName="main"
Jul 29 04:56:50.489708 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:50.489688    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e8f64fc-3471-4636-a5a3-dcdbb65b6a73" containerName="main"
Jul 29 04:56:50.489784 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:50.489753    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e8f64fc-3471-4636-a5a3-dcdbb65b6a73" containerName="main"
Jul 29 04:56:50.492770 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:50.492752    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7677-140502348540064"
Jul 29 04:56:50.495551 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:50.495526    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-k8w2g\""
Jul 29 04:56:50.499174 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:50.499155    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7677-140502348540064"]
Jul 29 04:56:50.593196 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:50.593170    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5zll\" (UniqueName: \"kubernetes.io/projected/de24a921-445c-436a-a175-e8825cf52d8c-kube-api-access-l5zll\") pod \"test-curl-7677-140502348540064\" (UID: \"de24a921-445c-436a-a175-e8825cf52d8c\") " pod="opendatahub/test-curl-7677-140502348540064"
Jul 29 04:56:50.693993 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:50.693954    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5zll\" (UniqueName: \"kubernetes.io/projected/de24a921-445c-436a-a175-e8825cf52d8c-kube-api-access-l5zll\") pod \"test-curl-7677-140502348540064\" (UID: \"de24a921-445c-436a-a175-e8825cf52d8c\") " pod="opendatahub/test-curl-7677-140502348540064"
Jul 29 04:56:50.702554 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:50.702536    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5zll\" (UniqueName: \"kubernetes.io/projected/de24a921-445c-436a-a175-e8825cf52d8c-kube-api-access-l5zll\") pod \"test-curl-7677-140502348540064\" (UID: \"de24a921-445c-436a-a175-e8825cf52d8c\") " pod="opendatahub/test-curl-7677-140502348540064"
Jul 29 04:56:50.803041 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:50.802974    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7677-140502348540064"
Jul 29 04:56:50.937393 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:50.937366    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7677-140502348540064"]
Jul 29 04:56:50.938534 ip-10-0-135-138 kubenswrapper[2570]: W0729 04:56:50.938511    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podde24a921_445c_436a_a175_e8825cf52d8c.slice/crio-71f21d98b0fb3a39ed218d29566f983ab38d9aa1fc2e74189183ade00f188dfb WatchSource:0}: Error finding container 71f21d98b0fb3a39ed218d29566f983ab38d9aa1fc2e74189183ade00f188dfb: Status 404 returned error can't find the container with id 71f21d98b0fb3a39ed218d29566f983ab38d9aa1fc2e74189183ade00f188dfb
Jul 29 04:56:50.939775 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:50.939759    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 04:56:51.117095 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:51.117063    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7677-140502348540064" event={"ID":"de24a921-445c-436a-a175-e8825cf52d8c","Type":"ContainerStarted","Data":"71f21d98b0fb3a39ed218d29566f983ab38d9aa1fc2e74189183ade00f188dfb"}
Jul 29 04:56:51.166079 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:51.166044    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-b70f7c/rate-test-model-b70f7c-kserve-7c4d9df799-qr8p2" podUID="75a76190-e96a-46bb-bd22-da051039ebb3" containerName="main" containerID="cri-o://e50efb917af604095e1ef4491887baa07e3b45bda40884116123ae4502218749" gracePeriod=45
Jul 29 04:56:51.593440 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:51.593419    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b70f7c/rate-test-model-b70f7c-kserve-7c4d9df799-qr8p2"
Jul 29 04:56:51.703778 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:51.703754    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75a76190-e96a-46bb-bd22-da051039ebb3-dshm\") pod \"75a76190-e96a-46bb-bd22-da051039ebb3\" (UID: \"75a76190-e96a-46bb-bd22-da051039ebb3\") "
Jul 29 04:56:51.703901 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:51.703792    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75a76190-e96a-46bb-bd22-da051039ebb3-home\") pod \"75a76190-e96a-46bb-bd22-da051039ebb3\" (UID: \"75a76190-e96a-46bb-bd22-da051039ebb3\") "
Jul 29 04:56:51.703901 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:51.703880    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75a76190-e96a-46bb-bd22-da051039ebb3-tmp-dir\") pod \"75a76190-e96a-46bb-bd22-da051039ebb3\" (UID: \"75a76190-e96a-46bb-bd22-da051039ebb3\") "
Jul 29 04:56:51.704020 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:51.703943    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75a76190-e96a-46bb-bd22-da051039ebb3-tls-certs\") pod \"75a76190-e96a-46bb-bd22-da051039ebb3\" (UID: \"75a76190-e96a-46bb-bd22-da051039ebb3\") "
Jul 29 04:56:51.704020 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:51.703980    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75a76190-e96a-46bb-bd22-da051039ebb3-model-cache\") pod \"75a76190-e96a-46bb-bd22-da051039ebb3\" (UID: \"75a76190-e96a-46bb-bd22-da051039ebb3\") "
Jul 29 04:56:51.704020 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:51.704015    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8nwtx\" (UniqueName: \"kubernetes.io/projected/75a76190-e96a-46bb-bd22-da051039ebb3-kube-api-access-8nwtx\") pod \"75a76190-e96a-46bb-bd22-da051039ebb3\" (UID: \"75a76190-e96a-46bb-bd22-da051039ebb3\") "
Jul 29 04:56:51.704175 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:51.704127    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75a76190-e96a-46bb-bd22-da051039ebb3-home" (OuterVolumeSpecName: "home") pod "75a76190-e96a-46bb-bd22-da051039ebb3" (UID: "75a76190-e96a-46bb-bd22-da051039ebb3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 04:56:51.704487 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:51.704363    2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75a76190-e96a-46bb-bd22-da051039ebb3-home\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:56:51.704593 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:51.704570    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75a76190-e96a-46bb-bd22-da051039ebb3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "75a76190-e96a-46bb-bd22-da051039ebb3" (UID: "75a76190-e96a-46bb-bd22-da051039ebb3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 04:56:51.704653 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:51.704633    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75a76190-e96a-46bb-bd22-da051039ebb3-model-cache" (OuterVolumeSpecName: "model-cache") pod "75a76190-e96a-46bb-bd22-da051039ebb3" (UID: "75a76190-e96a-46bb-bd22-da051039ebb3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 04:56:51.705859 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:51.705833    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75a76190-e96a-46bb-bd22-da051039ebb3-dshm" (OuterVolumeSpecName: "dshm") pod "75a76190-e96a-46bb-bd22-da051039ebb3" (UID: "75a76190-e96a-46bb-bd22-da051039ebb3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 04:56:51.706577 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:51.706556    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75a76190-e96a-46bb-bd22-da051039ebb3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "75a76190-e96a-46bb-bd22-da051039ebb3" (UID: "75a76190-e96a-46bb-bd22-da051039ebb3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 04:56:51.706671 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:51.706646    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75a76190-e96a-46bb-bd22-da051039ebb3-kube-api-access-8nwtx" (OuterVolumeSpecName: "kube-api-access-8nwtx") pod "75a76190-e96a-46bb-bd22-da051039ebb3" (UID: "75a76190-e96a-46bb-bd22-da051039ebb3"). InnerVolumeSpecName "kube-api-access-8nwtx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 04:56:51.805688 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:51.805658    2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75a76190-e96a-46bb-bd22-da051039ebb3-dshm\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:56:51.805688 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:51.805686    2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75a76190-e96a-46bb-bd22-da051039ebb3-tmp-dir\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:56:51.805893 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:51.805697    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75a76190-e96a-46bb-bd22-da051039ebb3-tls-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:56:51.805893 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:51.805704    2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75a76190-e96a-46bb-bd22-da051039ebb3-model-cache\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:56:51.805893 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:51.805712    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nwtx\" (UniqueName: \"kubernetes.io/projected/75a76190-e96a-46bb-bd22-da051039ebb3-kube-api-access-8nwtx\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:56:52.123955 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:52.123877    2570 generic.go:356] "Generic (PLEG): container finished" podID="de24a921-445c-436a-a175-e8825cf52d8c" containerID="2329c01bc08feda4ffe765434772224eaaafc4a061077874f2c84942685c0a3f" exitCode=0
Jul 29 04:56:52.124100 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:52.123967    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7677-140502348540064" event={"ID":"de24a921-445c-436a-a175-e8825cf52d8c","Type":"ContainerDied","Data":"2329c01bc08feda4ffe765434772224eaaafc4a061077874f2c84942685c0a3f"}
Jul 29 04:56:52.125092 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:52.125074    2570 generic.go:356] "Generic (PLEG): container finished" podID="75a76190-e96a-46bb-bd22-da051039ebb3" containerID="e50efb917af604095e1ef4491887baa07e3b45bda40884116123ae4502218749" exitCode=0
Jul 29 04:56:52.125169 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:52.125103    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b70f7c/rate-test-model-b70f7c-kserve-7c4d9df799-qr8p2" event={"ID":"75a76190-e96a-46bb-bd22-da051039ebb3","Type":"ContainerDied","Data":"e50efb917af604095e1ef4491887baa07e3b45bda40884116123ae4502218749"}
Jul 29 04:56:52.125169 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:52.125138    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b70f7c/rate-test-model-b70f7c-kserve-7c4d9df799-qr8p2" event={"ID":"75a76190-e96a-46bb-bd22-da051039ebb3","Type":"ContainerDied","Data":"46a3db83f8754e99b343fd320b159288d26eb8646ef72e3202a132d4197cbef6"}
Jul 29 04:56:52.125169 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:52.125155    2570 scope.go:117] "RemoveContainer" containerID="e50efb917af604095e1ef4491887baa07e3b45bda40884116123ae4502218749"
Jul 29 04:56:52.125274 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:52.125107    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b70f7c/rate-test-model-b70f7c-kserve-7c4d9df799-qr8p2"
Jul 29 04:56:52.133865 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:52.133848    2570 scope.go:117] "RemoveContainer" containerID="e50efb917af604095e1ef4491887baa07e3b45bda40884116123ae4502218749"
Jul 29 04:56:52.134154 ip-10-0-135-138 kubenswrapper[2570]: E0729 04:56:52.134136    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e50efb917af604095e1ef4491887baa07e3b45bda40884116123ae4502218749\": container with ID starting with e50efb917af604095e1ef4491887baa07e3b45bda40884116123ae4502218749 not found: ID does not exist" containerID="e50efb917af604095e1ef4491887baa07e3b45bda40884116123ae4502218749"
Jul 29 04:56:52.134213 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:52.134162    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e50efb917af604095e1ef4491887baa07e3b45bda40884116123ae4502218749"} err="failed to get container status \"e50efb917af604095e1ef4491887baa07e3b45bda40884116123ae4502218749\": rpc error: code = NotFound desc = could not find container \"e50efb917af604095e1ef4491887baa07e3b45bda40884116123ae4502218749\": container with ID starting with e50efb917af604095e1ef4491887baa07e3b45bda40884116123ae4502218749 not found: ID does not exist"
Jul 29 04:56:52.152097 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:52.152074    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-b70f7c/rate-test-model-b70f7c-kserve-7c4d9df799-qr8p2"]
Jul 29 04:56:52.155431 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:52.155409    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-b70f7c/rate-test-model-b70f7c-kserve-7c4d9df799-qr8p2"]
Jul 29 04:56:53.263523 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:53.263497    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7677-140502348540064"
Jul 29 04:56:53.316728 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:53.316701    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7677-140502348540064_de24a921-445c-436a-a175-e8825cf52d8c/test-curl-7677-140502348540064/0.log"
Jul 29 04:56:53.318890 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:53.318872    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l5zll\" (UniqueName: \"kubernetes.io/projected/de24a921-445c-436a-a175-e8825cf52d8c-kube-api-access-l5zll\") pod \"de24a921-445c-436a-a175-e8825cf52d8c\" (UID: \"de24a921-445c-436a-a175-e8825cf52d8c\") "
Jul 29 04:56:53.321364 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:53.321342    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de24a921-445c-436a-a175-e8825cf52d8c-kube-api-access-l5zll" (OuterVolumeSpecName: "kube-api-access-l5zll") pod "de24a921-445c-436a-a175-e8825cf52d8c" (UID: "de24a921-445c-436a-a175-e8825cf52d8c"). InnerVolumeSpecName "kube-api-access-l5zll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 04:56:53.344062 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:53.344036    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7677-140502348540064"]
Jul 29 04:56:53.357769 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:53.357747    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7677-140502348540064"]
Jul 29 04:56:53.419985 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:53.419933    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5zll\" (UniqueName: \"kubernetes.io/projected/de24a921-445c-436a-a175-e8825cf52d8c-kube-api-access-l5zll\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:56:53.491460 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:53.491434    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7677-140502348540064"]
Jul 29 04:56:53.491870 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:53.491856    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75a76190-e96a-46bb-bd22-da051039ebb3" containerName="main"
Jul 29 04:56:53.491930 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:53.491872    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="75a76190-e96a-46bb-bd22-da051039ebb3" containerName="main"
Jul 29 04:56:53.491930 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:53.491895    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de24a921-445c-436a-a175-e8825cf52d8c" containerName="test-curl-7677-140502348540064"
Jul 29 04:56:53.491930 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:53.491901    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="de24a921-445c-436a-a175-e8825cf52d8c" containerName="test-curl-7677-140502348540064"
Jul 29 04:56:53.492043 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:53.491972    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="de24a921-445c-436a-a175-e8825cf52d8c" containerName="test-curl-7677-140502348540064"
Jul 29 04:56:53.492043 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:53.491981    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="75a76190-e96a-46bb-bd22-da051039ebb3" containerName="main"
Jul 29 04:56:53.495028 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:53.495011    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7677-140502348540064"
Jul 29 04:56:53.502622 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:53.502601    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7677-140502348540064"]
Jul 29 04:56:53.520412 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:53.520379    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p899n\" (UniqueName: \"kubernetes.io/projected/665fa7ad-a41d-4590-923a-b08c77233dea-kube-api-access-p899n\") pod \"test-curl-7677-140502348540064\" (UID: \"665fa7ad-a41d-4590-923a-b08c77233dea\") " pod="opendatahub/test-curl-7677-140502348540064"
Jul 29 04:56:53.621566 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:53.621540    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p899n\" (UniqueName: \"kubernetes.io/projected/665fa7ad-a41d-4590-923a-b08c77233dea-kube-api-access-p899n\") pod \"test-curl-7677-140502348540064\" (UID: \"665fa7ad-a41d-4590-923a-b08c77233dea\") " pod="opendatahub/test-curl-7677-140502348540064"
Jul 29 04:56:53.629801 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:53.629772    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p899n\" (UniqueName: \"kubernetes.io/projected/665fa7ad-a41d-4590-923a-b08c77233dea-kube-api-access-p899n\") pod \"test-curl-7677-140502348540064\" (UID: \"665fa7ad-a41d-4590-923a-b08c77233dea\") " pod="opendatahub/test-curl-7677-140502348540064"
Jul 29 04:56:53.808015 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:53.807945    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7677-140502348540064"
Jul 29 04:56:53.846924 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:53.846897    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75a76190-e96a-46bb-bd22-da051039ebb3" path="/var/lib/kubelet/pods/75a76190-e96a-46bb-bd22-da051039ebb3/volumes"
Jul 29 04:56:53.847241 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:53.847225    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de24a921-445c-436a-a175-e8825cf52d8c" path="/var/lib/kubelet/pods/de24a921-445c-436a-a175-e8825cf52d8c/volumes"
Jul 29 04:56:53.941658 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:53.941628    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7677-140502348540064"]
Jul 29 04:56:53.945023 ip-10-0-135-138 kubenswrapper[2570]: W0729 04:56:53.944996    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod665fa7ad_a41d_4590_923a_b08c77233dea.slice/crio-e83e0e2524948fb59f523fb741a0969f6cf38296fc5fffdc24268492a6091c3b WatchSource:0}: Error finding container e83e0e2524948fb59f523fb741a0969f6cf38296fc5fffdc24268492a6091c3b: Status 404 returned error can't find the container with id e83e0e2524948fb59f523fb741a0969f6cf38296fc5fffdc24268492a6091c3b
Jul 29 04:56:54.136483 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:54.136447    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7677-140502348540064" event={"ID":"665fa7ad-a41d-4590-923a-b08c77233dea","Type":"ContainerStarted","Data":"e83e0e2524948fb59f523fb741a0969f6cf38296fc5fffdc24268492a6091c3b"}
Jul 29 04:56:54.137768 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:54.137753    2570 scope.go:117] "RemoveContainer" containerID="2329c01bc08feda4ffe765434772224eaaafc4a061077874f2c84942685c0a3f"
Jul 29 04:56:54.137768 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:54.137763    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7677-140502348540064"
Jul 29 04:56:55.145346 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:55.145308    2570 generic.go:356] "Generic (PLEG): container finished" podID="665fa7ad-a41d-4590-923a-b08c77233dea" containerID="68d3c3d6a59af2eaddc652c9f4b030027984c01048b084544adcdcece33b51ad" exitCode=0
Jul 29 04:56:55.145346 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:55.145350    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7677-140502348540064" event={"ID":"665fa7ad-a41d-4590-923a-b08c77233dea","Type":"ContainerDied","Data":"68d3c3d6a59af2eaddc652c9f4b030027984c01048b084544adcdcece33b51ad"}
Jul 29 04:56:56.290204 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:56.290184    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7677-140502348540064"
Jul 29 04:56:56.310932 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:56.310907    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7677-140502348540064_665fa7ad-a41d-4590-923a-b08c77233dea/test-curl-7677-140502348540064/0.log"
Jul 29 04:56:56.332880 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:56.332852    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7677-140502348540064"]
Jul 29 04:56:56.339237 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:56.339215    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7677-140502348540064"]
Jul 29 04:56:56.344584 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:56.344565    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p899n\" (UniqueName: \"kubernetes.io/projected/665fa7ad-a41d-4590-923a-b08c77233dea-kube-api-access-p899n\") pod \"665fa7ad-a41d-4590-923a-b08c77233dea\" (UID: \"665fa7ad-a41d-4590-923a-b08c77233dea\") "
Jul 29 04:56:56.346686 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:56.346664    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/665fa7ad-a41d-4590-923a-b08c77233dea-kube-api-access-p899n" (OuterVolumeSpecName: "kube-api-access-p899n") pod "665fa7ad-a41d-4590-923a-b08c77233dea" (UID: "665fa7ad-a41d-4590-923a-b08c77233dea"). InnerVolumeSpecName "kube-api-access-p899n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 04:56:56.445496 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:56.445421    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p899n\" (UniqueName: \"kubernetes.io/projected/665fa7ad-a41d-4590-923a-b08c77233dea-kube-api-access-p899n\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:56:57.158590 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:57.158557    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7677-140502348540064"
Jul 29 04:56:57.158590 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:57.158589    2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e83e0e2524948fb59f523fb741a0969f6cf38296fc5fffdc24268492a6091c3b"
Jul 29 04:56:57.168859 ip-10-0-135-138 kubenswrapper[2570]: E0729 04:56:57.168812    2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7677-140502348540064\" is forbidden: User \"system:node:ip-10-0-135-138.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-138.ec2.internal' and this object" podUID="665fa7ad-a41d-4590-923a-b08c77233dea" pod="opendatahub/test-curl-7677-140502348540064"
Jul 29 04:56:57.834995 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:56:57.834962    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="665fa7ad-a41d-4590-923a-b08c77233dea" path="/var/lib/kubelet/pods/665fa7ad-a41d-4590-923a-b08c77233dea/volumes"
Jul 29 04:57:00.259005 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:00.258971    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ec7357-7d7bcdf6b-fxs5l"]
Jul 29 04:57:00.259479 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:00.259461    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="665fa7ad-a41d-4590-923a-b08c77233dea" containerName="test-curl-7677-140502348540064"
Jul 29 04:57:00.259479 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:00.259480    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="665fa7ad-a41d-4590-923a-b08c77233dea" containerName="test-curl-7677-140502348540064"
Jul 29 04:57:00.259621 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:00.259591    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="665fa7ad-a41d-4590-923a-b08c77233dea" containerName="test-curl-7677-140502348540064"
Jul 29 04:57:00.262857 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:00.262838    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ec7357-7d7bcdf6b-fxs5l"
Jul 29 04:57:00.265955 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:00.265932    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-ec7357-serving-cert\""
Jul 29 04:57:00.272535 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:00.272504    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ec7357-7d7bcdf6b-fxs5l"]
Jul 29 04:57:00.380571 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:00.380541    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a283dca-805f-49b3-93ec-310c2ec07c1d-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ec7357-7d7bcdf6b-fxs5l\" (UID: \"5a283dca-805f-49b3-93ec-310c2ec07c1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ec7357-7d7bcdf6b-fxs5l"
Jul 29 04:57:00.380732 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:00.380601    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49f2r\" (UniqueName: \"kubernetes.io/projected/5a283dca-805f-49b3-93ec-310c2ec07c1d-kube-api-access-49f2r\") pod \"maas-api-e2e-ipp-a-ec7357-7d7bcdf6b-fxs5l\" (UID: \"5a283dca-805f-49b3-93ec-310c2ec07c1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ec7357-7d7bcdf6b-fxs5l"
Jul 29 04:57:00.380732 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:00.380680    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a283dca-805f-49b3-93ec-310c2ec07c1d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ec7357-7d7bcdf6b-fxs5l\" (UID: \"5a283dca-805f-49b3-93ec-310c2ec07c1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ec7357-7d7bcdf6b-fxs5l"
Jul 29 04:57:00.482131 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:00.482103    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49f2r\" (UniqueName: \"kubernetes.io/projected/5a283dca-805f-49b3-93ec-310c2ec07c1d-kube-api-access-49f2r\") pod \"maas-api-e2e-ipp-a-ec7357-7d7bcdf6b-fxs5l\" (UID: \"5a283dca-805f-49b3-93ec-310c2ec07c1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ec7357-7d7bcdf6b-fxs5l"
Jul 29 04:57:00.482280 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:00.482150    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a283dca-805f-49b3-93ec-310c2ec07c1d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ec7357-7d7bcdf6b-fxs5l\" (UID: \"5a283dca-805f-49b3-93ec-310c2ec07c1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ec7357-7d7bcdf6b-fxs5l"
Jul 29 04:57:00.482280 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:00.482213    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a283dca-805f-49b3-93ec-310c2ec07c1d-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ec7357-7d7bcdf6b-fxs5l\" (UID: \"5a283dca-805f-49b3-93ec-310c2ec07c1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ec7357-7d7bcdf6b-fxs5l"
Jul 29 04:57:00.482979 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:00.482956    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a283dca-805f-49b3-93ec-310c2ec07c1d-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ec7357-7d7bcdf6b-fxs5l\" (UID: \"5a283dca-805f-49b3-93ec-310c2ec07c1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ec7357-7d7bcdf6b-fxs5l"
Jul 29 04:57:00.484881 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:00.484863    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a283dca-805f-49b3-93ec-310c2ec07c1d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ec7357-7d7bcdf6b-fxs5l\" (UID: \"5a283dca-805f-49b3-93ec-310c2ec07c1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ec7357-7d7bcdf6b-fxs5l"
Jul 29 04:57:00.495274 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:00.495252    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49f2r\" (UniqueName: \"kubernetes.io/projected/5a283dca-805f-49b3-93ec-310c2ec07c1d-kube-api-access-49f2r\") pod \"maas-api-e2e-ipp-a-ec7357-7d7bcdf6b-fxs5l\" (UID: \"5a283dca-805f-49b3-93ec-310c2ec07c1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ec7357-7d7bcdf6b-fxs5l"
Jul 29 04:57:00.575184 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:00.575109    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ec7357-7d7bcdf6b-fxs5l"
Jul 29 04:57:00.718801 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:00.718767    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ec7357-7d7bcdf6b-fxs5l"]
Jul 29 04:57:00.720969 ip-10-0-135-138 kubenswrapper[2570]: W0729 04:57:00.720941    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a283dca_805f_49b3_93ec_310c2ec07c1d.slice/crio-5201c14472d43af7ffbc14383927ba44e1e2b79e8626de45b241fcecda86a18a WatchSource:0}: Error finding container 5201c14472d43af7ffbc14383927ba44e1e2b79e8626de45b241fcecda86a18a: Status 404 returned error can't find the container with id 5201c14472d43af7ffbc14383927ba44e1e2b79e8626de45b241fcecda86a18a
Jul 29 04:57:00.971308 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:00.971274    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ec7357-69f47849d8-kqpvh"]
Jul 29 04:57:00.976226 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:00.976210    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec7357-69f47849d8-kqpvh"
Jul 29 04:57:00.979134 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:00.979105    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-ec7357-dockercfg-wbn2c\""
Jul 29 04:57:00.979382 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:00.979366    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-ec7357\""
Jul 29 04:57:00.986710 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:00.986688    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ec7357-69f47849d8-kqpvh"]
Jul 29 04:57:01.008918 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:01.008890    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ec7357-67c6b5c4f4-l5nrc"]
Jul 29 04:57:01.012193 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:01.012174    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ec7357-67c6b5c4f4-l5nrc"
Jul 29 04:57:01.030382 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:01.030361    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ec7357-67c6b5c4f4-l5nrc"]
Jul 29 04:57:01.087178 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:01.087147    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b11c4ff-64ad-44bc-9ea2-df83a32ae814-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ec7357-67c6b5c4f4-l5nrc\" (UID: \"4b11c4ff-64ad-44bc-9ea2-df83a32ae814\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ec7357-67c6b5c4f4-l5nrc"
Jul 29 04:57:01.087321 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:01.087233    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hmqg\" (UniqueName: \"kubernetes.io/projected/4b11c4ff-64ad-44bc-9ea2-df83a32ae814-kube-api-access-4hmqg\") pod \"payload-processing-e2e-ipp-a-ec7357-67c6b5c4f4-l5nrc\" (UID: \"4b11c4ff-64ad-44bc-9ea2-df83a32ae814\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ec7357-67c6b5c4f4-l5nrc"
Jul 29 04:57:01.087321 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:01.087262    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea001ef5-473d-4e0b-9437-eabb4094553d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ec7357-69f47849d8-kqpvh\" (UID: \"ea001ef5-473d-4e0b-9437-eabb4094553d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec7357-69f47849d8-kqpvh"
Jul 29 04:57:01.087321 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:01.087292    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw2fz\" (UniqueName: \"kubernetes.io/projected/ea001ef5-473d-4e0b-9437-eabb4094553d-kube-api-access-jw2fz\") pod \"payload-pre-processing-e2e-ipp-a-ec7357-69f47849d8-kqpvh\" (UID: \"ea001ef5-473d-4e0b-9437-eabb4094553d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec7357-69f47849d8-kqpvh"
Jul 29 04:57:01.180995 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:01.180970    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ec7357-7d7bcdf6b-fxs5l" event={"ID":"5a283dca-805f-49b3-93ec-310c2ec07c1d","Type":"ContainerStarted","Data":"5201c14472d43af7ffbc14383927ba44e1e2b79e8626de45b241fcecda86a18a"}
Jul 29 04:57:01.188501 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:01.188477    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b11c4ff-64ad-44bc-9ea2-df83a32ae814-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ec7357-67c6b5c4f4-l5nrc\" (UID: \"4b11c4ff-64ad-44bc-9ea2-df83a32ae814\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ec7357-67c6b5c4f4-l5nrc"
Jul 29 04:57:01.188599 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:01.188539    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hmqg\" (UniqueName: \"kubernetes.io/projected/4b11c4ff-64ad-44bc-9ea2-df83a32ae814-kube-api-access-4hmqg\") pod \"payload-processing-e2e-ipp-a-ec7357-67c6b5c4f4-l5nrc\" (UID: \"4b11c4ff-64ad-44bc-9ea2-df83a32ae814\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ec7357-67c6b5c4f4-l5nrc"
Jul 29 04:57:01.188599 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:01.188579    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea001ef5-473d-4e0b-9437-eabb4094553d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ec7357-69f47849d8-kqpvh\" (UID: \"ea001ef5-473d-4e0b-9437-eabb4094553d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec7357-69f47849d8-kqpvh"
Jul 29 04:57:01.188726 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:01.188606    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw2fz\" (UniqueName: \"kubernetes.io/projected/ea001ef5-473d-4e0b-9437-eabb4094553d-kube-api-access-jw2fz\") pod \"payload-pre-processing-e2e-ipp-a-ec7357-69f47849d8-kqpvh\" (UID: \"ea001ef5-473d-4e0b-9437-eabb4094553d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec7357-69f47849d8-kqpvh"
Jul 29 04:57:01.189202 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:01.189181    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b11c4ff-64ad-44bc-9ea2-df83a32ae814-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ec7357-67c6b5c4f4-l5nrc\" (UID: \"4b11c4ff-64ad-44bc-9ea2-df83a32ae814\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ec7357-67c6b5c4f4-l5nrc"
Jul 29 04:57:01.189365 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:01.189345    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea001ef5-473d-4e0b-9437-eabb4094553d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ec7357-69f47849d8-kqpvh\" (UID: \"ea001ef5-473d-4e0b-9437-eabb4094553d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec7357-69f47849d8-kqpvh"
Jul 29 04:57:01.197856 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:01.197814    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw2fz\" (UniqueName: \"kubernetes.io/projected/ea001ef5-473d-4e0b-9437-eabb4094553d-kube-api-access-jw2fz\") pod \"payload-pre-processing-e2e-ipp-a-ec7357-69f47849d8-kqpvh\" (UID: \"ea001ef5-473d-4e0b-9437-eabb4094553d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec7357-69f47849d8-kqpvh"
Jul 29 04:57:01.197937 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:01.197919    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hmqg\" (UniqueName: \"kubernetes.io/projected/4b11c4ff-64ad-44bc-9ea2-df83a32ae814-kube-api-access-4hmqg\") pod \"payload-processing-e2e-ipp-a-ec7357-67c6b5c4f4-l5nrc\" (UID: \"4b11c4ff-64ad-44bc-9ea2-df83a32ae814\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ec7357-67c6b5c4f4-l5nrc"
Jul 29 04:57:01.288449 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:01.288420    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec7357-69f47849d8-kqpvh"
Jul 29 04:57:01.322198 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:01.322168    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ec7357-67c6b5c4f4-l5nrc"
Jul 29 04:57:01.441875 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:01.441848    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ec7357-69f47849d8-kqpvh"]
Jul 29 04:57:01.442482 ip-10-0-135-138 kubenswrapper[2570]: W0729 04:57:01.442457    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea001ef5_473d_4e0b_9437_eabb4094553d.slice/crio-9bcc22d5a7c90f17a56c615656e0e698810821deb478ba2f57bffa5f4fa278bc WatchSource:0}: Error finding container 9bcc22d5a7c90f17a56c615656e0e698810821deb478ba2f57bffa5f4fa278bc: Status 404 returned error can't find the container with id 9bcc22d5a7c90f17a56c615656e0e698810821deb478ba2f57bffa5f4fa278bc
Jul 29 04:57:01.492444 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:01.492389    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ec7357-67c6b5c4f4-l5nrc"]
Jul 29 04:57:01.495169 ip-10-0-135-138 kubenswrapper[2570]: W0729 04:57:01.495145    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b11c4ff_64ad_44bc_9ea2_df83a32ae814.slice/crio-694f0201ff85f4af1b3331dac9aa5192ddaeb494758807f33121f49a6232eb43 WatchSource:0}: Error finding container 694f0201ff85f4af1b3331dac9aa5192ddaeb494758807f33121f49a6232eb43: Status 404 returned error can't find the container with id 694f0201ff85f4af1b3331dac9aa5192ddaeb494758807f33121f49a6232eb43
Jul 29 04:57:02.187589 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:02.187555    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ec7357-67c6b5c4f4-l5nrc" event={"ID":"4b11c4ff-64ad-44bc-9ea2-df83a32ae814","Type":"ContainerStarted","Data":"d9ee6f93250669528ebee75190eafa1009cacb857408df1f2c3aa7add5cb01e6"}
Jul 29 04:57:02.187752 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:02.187605    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ec7357-67c6b5c4f4-l5nrc" event={"ID":"4b11c4ff-64ad-44bc-9ea2-df83a32ae814","Type":"ContainerStarted","Data":"694f0201ff85f4af1b3331dac9aa5192ddaeb494758807f33121f49a6232eb43"}
Jul 29 04:57:02.187752 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:02.187737    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ec7357-67c6b5c4f4-l5nrc"
Jul 29 04:57:02.188988 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:02.188958    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ec7357-7d7bcdf6b-fxs5l" event={"ID":"5a283dca-805f-49b3-93ec-310c2ec07c1d","Type":"ContainerStarted","Data":"00be6115785e799e9710e60ccde50e3d513672ead690af85f713ce9160a5e779"}
Jul 29 04:57:02.189109 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:02.189012    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ec7357-7d7bcdf6b-fxs5l"
Jul 29 04:57:02.190292 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:02.190275    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec7357-69f47849d8-kqpvh" event={"ID":"ea001ef5-473d-4e0b-9437-eabb4094553d","Type":"ContainerStarted","Data":"e1e4c00c6917329c94e4165a602e43fafbb36188784b7f7d135a0b0a9c46cc58"}
Jul 29 04:57:02.190380 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:02.190305    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec7357-69f47849d8-kqpvh" event={"ID":"ea001ef5-473d-4e0b-9437-eabb4094553d","Type":"ContainerStarted","Data":"9bcc22d5a7c90f17a56c615656e0e698810821deb478ba2f57bffa5f4fa278bc"}
Jul 29 04:57:02.190380 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:02.190344    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec7357-69f47849d8-kqpvh"
Jul 29 04:57:02.208448 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:02.208400    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-ec7357-67c6b5c4f4-l5nrc" podStartSLOduration=2.208384682 podStartE2EDuration="2.208384682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:57:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:57:02.205802911 +0000 UTC m=+3686.935934025" watchObservedRunningTime="2026-07-29 04:57:02.208384682 +0000 UTC m=+3686.938515755"
Jul 29 04:57:02.230080 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:02.230034    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ec7357-7d7bcdf6b-fxs5l" podStartSLOduration=1.776214726 podStartE2EDuration="2.230015609s" totalImagesPullingTime="453.800883ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:57:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:57:02.229189026 +0000 UTC m=+3686.959320097" watchObservedRunningTime="2026-07-29 04:57:02.230015609 +0000 UTC m=+3686.960146685"
Jul 29 04:57:02.248416 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:02.248379    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec7357-69f47849d8-kqpvh" podStartSLOduration=2.248366984 podStartE2EDuration="2.248366984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:57:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:57:02.247507431 +0000 UTC m=+3686.977638503" watchObservedRunningTime="2026-07-29 04:57:02.248366984 +0000 UTC m=+3686.978498054"
Jul 29 04:57:08.201423 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:08.201393    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ec7357-7d7bcdf6b-fxs5l"
Jul 29 04:57:13.198555 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:13.198527    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ec7357-67c6b5c4f4-l5nrc"
Jul 29 04:57:13.199036 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:13.198842    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec7357-69f47849d8-kqpvh"
Jul 29 04:57:32.747497 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:32.747465    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2cb3bf-77cf9bcfff-k67d8"]
Jul 29 04:57:32.751337 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:32.751320    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2cb3bf-77cf9bcfff-k67d8"
Jul 29 04:57:32.755692 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:32.755674    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-2cb3bf-serving-cert\""
Jul 29 04:57:32.763841 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:32.763805    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2cb3bf-77cf9bcfff-k67d8"]
Jul 29 04:57:32.784324 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:32.784300    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/182fdd57-ce2f-4ca0-acc3-a781eb32b6e5-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-2cb3bf-77cf9bcfff-k67d8\" (UID: \"182fdd57-ce2f-4ca0-acc3-a781eb32b6e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2cb3bf-77cf9bcfff-k67d8"
Jul 29 04:57:32.784426 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:32.784353    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzlkg\" (UniqueName: \"kubernetes.io/projected/182fdd57-ce2f-4ca0-acc3-a781eb32b6e5-kube-api-access-pzlkg\") pod \"maas-api-e2e-ipp-b-2cb3bf-77cf9bcfff-k67d8\" (UID: \"182fdd57-ce2f-4ca0-acc3-a781eb32b6e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2cb3bf-77cf9bcfff-k67d8"
Jul 29 04:57:32.784426 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:32.784418    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/182fdd57-ce2f-4ca0-acc3-a781eb32b6e5-maas-api-tls\") pod \"maas-api-e2e-ipp-b-2cb3bf-77cf9bcfff-k67d8\" (UID: \"182fdd57-ce2f-4ca0-acc3-a781eb32b6e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2cb3bf-77cf9bcfff-k67d8"
Jul 29 04:57:32.885260 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:32.885231    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzlkg\" (UniqueName: \"kubernetes.io/projected/182fdd57-ce2f-4ca0-acc3-a781eb32b6e5-kube-api-access-pzlkg\") pod \"maas-api-e2e-ipp-b-2cb3bf-77cf9bcfff-k67d8\" (UID: \"182fdd57-ce2f-4ca0-acc3-a781eb32b6e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2cb3bf-77cf9bcfff-k67d8"
Jul 29 04:57:32.885431 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:32.885282    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/182fdd57-ce2f-4ca0-acc3-a781eb32b6e5-maas-api-tls\") pod \"maas-api-e2e-ipp-b-2cb3bf-77cf9bcfff-k67d8\" (UID: \"182fdd57-ce2f-4ca0-acc3-a781eb32b6e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2cb3bf-77cf9bcfff-k67d8"
Jul 29 04:57:32.885431 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:32.885344    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/182fdd57-ce2f-4ca0-acc3-a781eb32b6e5-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-2cb3bf-77cf9bcfff-k67d8\" (UID: \"182fdd57-ce2f-4ca0-acc3-a781eb32b6e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2cb3bf-77cf9bcfff-k67d8"
Jul 29 04:57:32.885506 ip-10-0-135-138 kubenswrapper[2570]: E0729 04:57:32.885438    2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-2cb3bf-serving-cert: secret "maas-api-e2e-ipp-b-2cb3bf-serving-cert" not found
Jul 29 04:57:32.885506 ip-10-0-135-138 kubenswrapper[2570]: E0729 04:57:32.885491    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/182fdd57-ce2f-4ca0-acc3-a781eb32b6e5-maas-api-tls podName:182fdd57-ce2f-4ca0-acc3-a781eb32b6e5 nodeName:}" failed. No retries permitted until 2026-07-29 04:57:33.385476234 +0000 UTC m=+3718.115607281 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/182fdd57-ce2f-4ca0-acc3-a781eb32b6e5-maas-api-tls") pod "maas-api-e2e-ipp-b-2cb3bf-77cf9bcfff-k67d8" (UID: "182fdd57-ce2f-4ca0-acc3-a781eb32b6e5") : secret "maas-api-e2e-ipp-b-2cb3bf-serving-cert" not found
Jul 29 04:57:32.885949 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:32.885934    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/182fdd57-ce2f-4ca0-acc3-a781eb32b6e5-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-2cb3bf-77cf9bcfff-k67d8\" (UID: \"182fdd57-ce2f-4ca0-acc3-a781eb32b6e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2cb3bf-77cf9bcfff-k67d8"
Jul 29 04:57:32.894475 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:32.894455    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzlkg\" (UniqueName: \"kubernetes.io/projected/182fdd57-ce2f-4ca0-acc3-a781eb32b6e5-kube-api-access-pzlkg\") pod \"maas-api-e2e-ipp-b-2cb3bf-77cf9bcfff-k67d8\" (UID: \"182fdd57-ce2f-4ca0-acc3-a781eb32b6e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2cb3bf-77cf9bcfff-k67d8"
Jul 29 04:57:33.390879 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:33.390847    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/182fdd57-ce2f-4ca0-acc3-a781eb32b6e5-maas-api-tls\") pod \"maas-api-e2e-ipp-b-2cb3bf-77cf9bcfff-k67d8\" (UID: \"182fdd57-ce2f-4ca0-acc3-a781eb32b6e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2cb3bf-77cf9bcfff-k67d8"
Jul 29 04:57:33.393325 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:33.393303    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/182fdd57-ce2f-4ca0-acc3-a781eb32b6e5-maas-api-tls\") pod \"maas-api-e2e-ipp-b-2cb3bf-77cf9bcfff-k67d8\" (UID: \"182fdd57-ce2f-4ca0-acc3-a781eb32b6e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2cb3bf-77cf9bcfff-k67d8"
Jul 29 04:57:33.467154 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:33.467118    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2cb3bf-6749775684-rg6nk"]
Jul 29 04:57:33.471187 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:33.471166    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2cb3bf-6749775684-rg6nk"
Jul 29 04:57:33.473864 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:33.473810    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-2cb3bf\""
Jul 29 04:57:33.474003 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:33.473952    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-2cb3bf-dockercfg-chv2f\""
Jul 29 04:57:33.483478 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:33.483452    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2cb3bf-6749775684-rg6nk"]
Jul 29 04:57:33.492030 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:33.492008    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsfds\" (UniqueName: \"kubernetes.io/projected/77cdd15d-361a-494b-9295-ba5f3810f12c-kube-api-access-nsfds\") pod \"payload-pre-processing-e2e-ipp-b-2cb3bf-6749775684-rg6nk\" (UID: \"77cdd15d-361a-494b-9295-ba5f3810f12c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2cb3bf-6749775684-rg6nk"
Jul 29 04:57:33.492158 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:33.492058    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77cdd15d-361a-494b-9295-ba5f3810f12c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2cb3bf-6749775684-rg6nk\" (UID: \"77cdd15d-361a-494b-9295-ba5f3810f12c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2cb3bf-6749775684-rg6nk"
Jul 29 04:57:33.593346 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:33.593308    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsfds\" (UniqueName: \"kubernetes.io/projected/77cdd15d-361a-494b-9295-ba5f3810f12c-kube-api-access-nsfds\") pod \"payload-pre-processing-e2e-ipp-b-2cb3bf-6749775684-rg6nk\" (UID: \"77cdd15d-361a-494b-9295-ba5f3810f12c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2cb3bf-6749775684-rg6nk"
Jul 29 04:57:33.593521 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:33.593390    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77cdd15d-361a-494b-9295-ba5f3810f12c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2cb3bf-6749775684-rg6nk\" (UID: \"77cdd15d-361a-494b-9295-ba5f3810f12c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2cb3bf-6749775684-rg6nk"
Jul 29 04:57:33.594425 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:33.594405    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77cdd15d-361a-494b-9295-ba5f3810f12c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2cb3bf-6749775684-rg6nk\" (UID: \"77cdd15d-361a-494b-9295-ba5f3810f12c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2cb3bf-6749775684-rg6nk"
Jul 29 04:57:33.601234 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:33.601212    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsfds\" (UniqueName: \"kubernetes.io/projected/77cdd15d-361a-494b-9295-ba5f3810f12c-kube-api-access-nsfds\") pod \"payload-pre-processing-e2e-ipp-b-2cb3bf-6749775684-rg6nk\" (UID: \"77cdd15d-361a-494b-9295-ba5f3810f12c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2cb3bf-6749775684-rg6nk"
Jul 29 04:57:33.662067 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:33.662007    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2cb3bf-77cf9bcfff-k67d8"
Jul 29 04:57:33.783323 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:33.783295    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2cb3bf-6749775684-rg6nk"
Jul 29 04:57:33.800557 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:33.800531    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2cb3bf-77cf9bcfff-k67d8"]
Jul 29 04:57:33.801657 ip-10-0-135-138 kubenswrapper[2570]: W0729 04:57:33.801630    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod182fdd57_ce2f_4ca0_acc3_a781eb32b6e5.slice/crio-1457fad88ed68c3da13fd3e68c3c2fbda18ecc6877c871993ad5471dfdd03c60 WatchSource:0}: Error finding container 1457fad88ed68c3da13fd3e68c3c2fbda18ecc6877c871993ad5471dfdd03c60: Status 404 returned error can't find the container with id 1457fad88ed68c3da13fd3e68c3c2fbda18ecc6877c871993ad5471dfdd03c60
Jul 29 04:57:33.925200 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:33.925174    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2cb3bf-6749775684-rg6nk"]
Jul 29 04:57:33.927528 ip-10-0-135-138 kubenswrapper[2570]: W0729 04:57:33.927489    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77cdd15d_361a_494b_9295_ba5f3810f12c.slice/crio-72b50809fdc5d5cf13de7d06a5abefbf36c9a2df0224d5f5abc08b4547070164 WatchSource:0}: Error finding container 72b50809fdc5d5cf13de7d06a5abefbf36c9a2df0224d5f5abc08b4547070164: Status 404 returned error can't find the container with id 72b50809fdc5d5cf13de7d06a5abefbf36c9a2df0224d5f5abc08b4547070164
Jul 29 04:57:34.354247 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:34.354209    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2cb3bf-77cf9bcfff-k67d8" event={"ID":"182fdd57-ce2f-4ca0-acc3-a781eb32b6e5","Type":"ContainerStarted","Data":"2d245a8943376cbe54442abc4cfacd6d6231ca8bc274cf145f80782c3d17cc34"}
Jul 29 04:57:34.354424 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:34.354277    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2cb3bf-77cf9bcfff-k67d8" event={"ID":"182fdd57-ce2f-4ca0-acc3-a781eb32b6e5","Type":"ContainerStarted","Data":"1457fad88ed68c3da13fd3e68c3c2fbda18ecc6877c871993ad5471dfdd03c60"}
Jul 29 04:57:34.354424 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:34.354303    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2cb3bf-77cf9bcfff-k67d8"
Jul 29 04:57:34.355791 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:34.355765    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2cb3bf-6749775684-rg6nk" event={"ID":"77cdd15d-361a-494b-9295-ba5f3810f12c","Type":"ContainerStarted","Data":"ca7c4450c054bf7622fb662daa7c2632ce72b3c2728e010215cc5d222576f914"}
Jul 29 04:57:34.355916 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:34.355809    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2cb3bf-6749775684-rg6nk" event={"ID":"77cdd15d-361a-494b-9295-ba5f3810f12c","Type":"ContainerStarted","Data":"72b50809fdc5d5cf13de7d06a5abefbf36c9a2df0224d5f5abc08b4547070164"}
Jul 29 04:57:34.355916 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:34.355870    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2cb3bf-6749775684-rg6nk"
Jul 29 04:57:34.370984 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:34.370945    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2cb3bf-77cf9bcfff-k67d8" podStartSLOduration=2.081917914 podStartE2EDuration="2.370931218s" totalImagesPullingTime="289.013304ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:57:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:57:34.369664642 +0000 UTC m=+3719.099795735" watchObservedRunningTime="2026-07-29 04:57:34.370931218 +0000 UTC m=+3719.101062288"
Jul 29 04:57:34.384356 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:34.384318    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2cb3bf-6749775684-rg6nk" podStartSLOduration=1.384306375 podStartE2EDuration="1.384306375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:57:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:57:34.383998876 +0000 UTC m=+3719.114129947" watchObservedRunningTime="2026-07-29 04:57:34.384306375 +0000 UTC m=+3719.114437446"
Jul 29 04:57:40.367302 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:40.367275    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2cb3bf-77cf9bcfff-k67d8"
Jul 29 04:57:45.364000 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:57:45.363971    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2cb3bf-6749775684-rg6nk"
Jul 29 04:58:04.379831 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:58:04.379789    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-gznsm_a1ffbe6e-a508-4235-a696-01cd077b7423/payload-processing/0.log"
Jul 29 04:58:49.207548 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:58:49.207511    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-ec7357-67c6b5c4f4-l5nrc_4b11c4ff-64ad-44bc-9ea2-df83a32ae814/payload-processing/0.log"
Jul 29 04:58:49.328844 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:58:49.328796    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-gznsm_a1ffbe6e-a508-4235-a696-01cd077b7423/payload-processing/0.log"
Jul 29 04:58:53.516587 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:58:53.516537    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-79e09a-686c956cf6-jkdvg"]
Jul 29 04:58:53.520405 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:58:53.520379    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-79e09a-686c956cf6-jkdvg"
Jul 29 04:58:53.523029 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:58:53.523007    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-79e09a-dockercfg-5cks5\""
Jul 29 04:58:53.523171 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:58:53.523050    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-79e09a\""
Jul 29 04:58:53.528282 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:58:53.528259    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sv85\" (UniqueName: \"kubernetes.io/projected/26b88bda-8450-4ae1-b8ca-5ada2b32f3a3-kube-api-access-2sv85\") pod \"payload-pre-processing-e2e-ipp-cleanup-79e09a-686c956cf6-jkdvg\" (UID: \"26b88bda-8450-4ae1-b8ca-5ada2b32f3a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-79e09a-686c956cf6-jkdvg"
Jul 29 04:58:53.528506 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:58:53.528482    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26b88bda-8450-4ae1-b8ca-5ada2b32f3a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-79e09a-686c956cf6-jkdvg\" (UID: \"26b88bda-8450-4ae1-b8ca-5ada2b32f3a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-79e09a-686c956cf6-jkdvg"
Jul 29 04:58:53.530930 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:58:53.530911    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-79e09a-686c956cf6-jkdvg"]
Jul 29 04:58:53.629256 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:58:53.629212    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26b88bda-8450-4ae1-b8ca-5ada2b32f3a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-79e09a-686c956cf6-jkdvg\" (UID: \"26b88bda-8450-4ae1-b8ca-5ada2b32f3a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-79e09a-686c956cf6-jkdvg"
Jul 29 04:58:53.629426 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:58:53.629319    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sv85\" (UniqueName: \"kubernetes.io/projected/26b88bda-8450-4ae1-b8ca-5ada2b32f3a3-kube-api-access-2sv85\") pod \"payload-pre-processing-e2e-ipp-cleanup-79e09a-686c956cf6-jkdvg\" (UID: \"26b88bda-8450-4ae1-b8ca-5ada2b32f3a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-79e09a-686c956cf6-jkdvg"
Jul 29 04:58:53.629975 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:58:53.629955    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26b88bda-8450-4ae1-b8ca-5ada2b32f3a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-79e09a-686c956cf6-jkdvg\" (UID: \"26b88bda-8450-4ae1-b8ca-5ada2b32f3a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-79e09a-686c956cf6-jkdvg"
Jul 29 04:58:53.637548 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:58:53.637527    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sv85\" (UniqueName: \"kubernetes.io/projected/26b88bda-8450-4ae1-b8ca-5ada2b32f3a3-kube-api-access-2sv85\") pod \"payload-pre-processing-e2e-ipp-cleanup-79e09a-686c956cf6-jkdvg\" (UID: \"26b88bda-8450-4ae1-b8ca-5ada2b32f3a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-79e09a-686c956cf6-jkdvg"
Jul 29 04:58:53.831400 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:58:53.831376    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-79e09a-686c956cf6-jkdvg"
Jul 29 04:58:54.175855 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:58:54.175756    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-79e09a-686c956cf6-jkdvg"]
Jul 29 04:58:54.179323 ip-10-0-135-138 kubenswrapper[2570]: W0729 04:58:54.179293    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26b88bda_8450_4ae1_b8ca_5ada2b32f3a3.slice/crio-234391dbbe7fcd70d0a48ef628baa3cb4b1cc4b4975eea365b5306afd34535a7 WatchSource:0}: Error finding container 234391dbbe7fcd70d0a48ef628baa3cb4b1cc4b4975eea365b5306afd34535a7: Status 404 returned error can't find the container with id 234391dbbe7fcd70d0a48ef628baa3cb4b1cc4b4975eea365b5306afd34535a7
Jul 29 04:58:54.773647 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:58:54.773610    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-79e09a-686c956cf6-jkdvg" event={"ID":"26b88bda-8450-4ae1-b8ca-5ada2b32f3a3","Type":"ContainerStarted","Data":"8d1b2a277a1db3a814210ef05b7281cdacafd4bbe5a0547fe619fea42accfe30"}
Jul 29 04:58:54.774049 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:58:54.773670    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-79e09a-686c956cf6-jkdvg" event={"ID":"26b88bda-8450-4ae1-b8ca-5ada2b32f3a3","Type":"ContainerStarted","Data":"234391dbbe7fcd70d0a48ef628baa3cb4b1cc4b4975eea365b5306afd34535a7"}
Jul 29 04:58:54.774049 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:58:54.773711    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-79e09a-686c956cf6-jkdvg"
Jul 29 04:58:54.789403 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:58:54.789359    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-79e09a-686c956cf6-jkdvg" podStartSLOduration=1.789344743 podStartE2EDuration="1.789344743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:58:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:58:54.788208995 +0000 UTC m=+3799.518340065" watchObservedRunningTime="2026-07-29 04:58:54.789344743 +0000 UTC m=+3799.519475812"
Jul 29 04:59:05.780332 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:05.780295    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-79e09a-686c956cf6-jkdvg"
Jul 29 04:59:23.436971 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:23.436926    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-79e09a-686c956cf6-jkdvg"]
Jul 29 04:59:23.437470 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:23.437167    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-79e09a-686c956cf6-jkdvg" podUID="26b88bda-8450-4ae1-b8ca-5ada2b32f3a3" containerName="payload-pre-processing" containerID="cri-o://8d1b2a277a1db3a814210ef05b7281cdacafd4bbe5a0547fe619fea42accfe30" gracePeriod=30
Jul 29 04:59:23.687853 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:23.687780    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-79e09a-686c956cf6-jkdvg"
Jul 29 04:59:23.803997 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:23.803962    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2sv85\" (UniqueName: \"kubernetes.io/projected/26b88bda-8450-4ae1-b8ca-5ada2b32f3a3-kube-api-access-2sv85\") pod \"26b88bda-8450-4ae1-b8ca-5ada2b32f3a3\" (UID: \"26b88bda-8450-4ae1-b8ca-5ada2b32f3a3\") "
Jul 29 04:59:23.804168 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:23.804070    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26b88bda-8450-4ae1-b8ca-5ada2b32f3a3-plugins-config-volume\") pod \"26b88bda-8450-4ae1-b8ca-5ada2b32f3a3\" (UID: \"26b88bda-8450-4ae1-b8ca-5ada2b32f3a3\") "
Jul 29 04:59:23.804412 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:23.804392    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26b88bda-8450-4ae1-b8ca-5ada2b32f3a3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "26b88bda-8450-4ae1-b8ca-5ada2b32f3a3" (UID: "26b88bda-8450-4ae1-b8ca-5ada2b32f3a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 04:59:23.806200 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:23.806177    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26b88bda-8450-4ae1-b8ca-5ada2b32f3a3-kube-api-access-2sv85" (OuterVolumeSpecName: "kube-api-access-2sv85") pod "26b88bda-8450-4ae1-b8ca-5ada2b32f3a3" (UID: "26b88bda-8450-4ae1-b8ca-5ada2b32f3a3"). InnerVolumeSpecName "kube-api-access-2sv85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 04:59:23.905634 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:23.905609    2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26b88bda-8450-4ae1-b8ca-5ada2b32f3a3-plugins-config-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:59:23.905634 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:23.905633    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sv85\" (UniqueName: \"kubernetes.io/projected/26b88bda-8450-4ae1-b8ca-5ada2b32f3a3-kube-api-access-2sv85\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:59:23.923037 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:23.923007    2570 generic.go:356] "Generic (PLEG): container finished" podID="26b88bda-8450-4ae1-b8ca-5ada2b32f3a3" containerID="8d1b2a277a1db3a814210ef05b7281cdacafd4bbe5a0547fe619fea42accfe30" exitCode=0
Jul 29 04:59:23.923138 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:23.923054    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-79e09a-686c956cf6-jkdvg"
Jul 29 04:59:23.923138 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:23.923091    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-79e09a-686c956cf6-jkdvg" event={"ID":"26b88bda-8450-4ae1-b8ca-5ada2b32f3a3","Type":"ContainerDied","Data":"8d1b2a277a1db3a814210ef05b7281cdacafd4bbe5a0547fe619fea42accfe30"}
Jul 29 04:59:23.923213 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:23.923147    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-79e09a-686c956cf6-jkdvg" event={"ID":"26b88bda-8450-4ae1-b8ca-5ada2b32f3a3","Type":"ContainerDied","Data":"234391dbbe7fcd70d0a48ef628baa3cb4b1cc4b4975eea365b5306afd34535a7"}
Jul 29 04:59:23.923213 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:23.923178    2570 scope.go:117] "RemoveContainer" containerID="8d1b2a277a1db3a814210ef05b7281cdacafd4bbe5a0547fe619fea42accfe30"
Jul 29 04:59:23.932443 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:23.932424    2570 scope.go:117] "RemoveContainer" containerID="8d1b2a277a1db3a814210ef05b7281cdacafd4bbe5a0547fe619fea42accfe30"
Jul 29 04:59:23.932696 ip-10-0-135-138 kubenswrapper[2570]: E0729 04:59:23.932679    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d1b2a277a1db3a814210ef05b7281cdacafd4bbe5a0547fe619fea42accfe30\": container with ID starting with 8d1b2a277a1db3a814210ef05b7281cdacafd4bbe5a0547fe619fea42accfe30 not found: ID does not exist" containerID="8d1b2a277a1db3a814210ef05b7281cdacafd4bbe5a0547fe619fea42accfe30"
Jul 29 04:59:23.932758 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:23.932709    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d1b2a277a1db3a814210ef05b7281cdacafd4bbe5a0547fe619fea42accfe30"} err="failed to get container status \"8d1b2a277a1db3a814210ef05b7281cdacafd4bbe5a0547fe619fea42accfe30\": rpc error: code = NotFound desc = could not find container \"8d1b2a277a1db3a814210ef05b7281cdacafd4bbe5a0547fe619fea42accfe30\": container with ID starting with 8d1b2a277a1db3a814210ef05b7281cdacafd4bbe5a0547fe619fea42accfe30 not found: ID does not exist"
Jul 29 04:59:23.939227 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:23.939180    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-79e09a-686c956cf6-jkdvg"]
Jul 29 04:59:23.942907 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:23.942882    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-79e09a-686c956cf6-jkdvg"]
Jul 29 04:59:25.843316 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:25.843275    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26b88bda-8450-4ae1-b8ca-5ada2b32f3a3" path="/var/lib/kubelet/pods/26b88bda-8450-4ae1-b8ca-5ada2b32f3a3/volumes"
Jul 29 04:59:33.418409 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:33.418377    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ec7357-f2gw6"]
Jul 29 04:59:33.418855 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:33.418842    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26b88bda-8450-4ae1-b8ca-5ada2b32f3a3" containerName="payload-pre-processing"
Jul 29 04:59:33.418855 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:33.418857    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="26b88bda-8450-4ae1-b8ca-5ada2b32f3a3" containerName="payload-pre-processing"
Jul 29 04:59:33.418943 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:33.418922    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="26b88bda-8450-4ae1-b8ca-5ada2b32f3a3" containerName="payload-pre-processing"
Jul 29 04:59:33.423445 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:33.423428    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ec7357-f2gw6"
Jul 29 04:59:33.426147 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:33.426122    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8frm\""
Jul 29 04:59:33.427712 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:33.427687    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ec7357-f2gw6"]
Jul 29 04:59:33.596080 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:33.596046    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/396e551e-00cf-4358-8f7f-1f2d56fd99a8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-ec7357-f2gw6\" (UID: \"396e551e-00cf-4358-8f7f-1f2d56fd99a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ec7357-f2gw6"
Jul 29 04:59:33.697215 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:33.697144    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/396e551e-00cf-4358-8f7f-1f2d56fd99a8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-ec7357-f2gw6\" (UID: \"396e551e-00cf-4358-8f7f-1f2d56fd99a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ec7357-f2gw6"
Jul 29 04:59:33.697738 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:33.697719    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/396e551e-00cf-4358-8f7f-1f2d56fd99a8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-ec7357-f2gw6\" (UID: \"396e551e-00cf-4358-8f7f-1f2d56fd99a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ec7357-f2gw6"
Jul 29 04:59:33.735111 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:33.735082    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ec7357-f2gw6"
Jul 29 04:59:33.874990 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:33.874967    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ec7357-f2gw6"]
Jul 29 04:59:33.876865 ip-10-0-135-138 kubenswrapper[2570]: W0729 04:59:33.876837    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod396e551e_00cf_4358_8f7f_1f2d56fd99a8.slice/crio-79c55dd1cb27812739d30fb6a40791cfe44a6f000a20bfea41facd00043993a2 WatchSource:0}: Error finding container 79c55dd1cb27812739d30fb6a40791cfe44a6f000a20bfea41facd00043993a2: Status 404 returned error can't find the container with id 79c55dd1cb27812739d30fb6a40791cfe44a6f000a20bfea41facd00043993a2
Jul 29 04:59:33.977602 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:33.977532    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ec7357-f2gw6" event={"ID":"396e551e-00cf-4358-8f7f-1f2d56fd99a8","Type":"ContainerStarted","Data":"79c55dd1cb27812739d30fb6a40791cfe44a6f000a20bfea41facd00043993a2"}
Jul 29 04:59:34.984084 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:34.984005    2570 generic.go:356] "Generic (PLEG): container finished" podID="396e551e-00cf-4358-8f7f-1f2d56fd99a8" containerID="b652926458675679112e4fa2da642b1527aa21c929a613e62c42bd8549ea9c8f" exitCode=0
Jul 29 04:59:34.984084 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:34.984045    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ec7357-f2gw6" event={"ID":"396e551e-00cf-4358-8f7f-1f2d56fd99a8","Type":"ContainerDied","Data":"b652926458675679112e4fa2da642b1527aa21c929a613e62c42bd8549ea9c8f"}
Jul 29 04:59:36.123737 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:36.123710    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ec7357-f2gw6"
Jul 29 04:59:36.224346 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:36.224311    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/396e551e-00cf-4358-8f7f-1f2d56fd99a8-maas-api-service-ca\") pod \"396e551e-00cf-4358-8f7f-1f2d56fd99a8\" (UID: \"396e551e-00cf-4358-8f7f-1f2d56fd99a8\") "
Jul 29 04:59:36.224679 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:36.224650    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/396e551e-00cf-4358-8f7f-1f2d56fd99a8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "396e551e-00cf-4358-8f7f-1f2d56fd99a8" (UID: "396e551e-00cf-4358-8f7f-1f2d56fd99a8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 04:59:36.325323 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:36.325303    2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/396e551e-00cf-4358-8f7f-1f2d56fd99a8-maas-api-service-ca\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:59:36.995774 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:36.995748    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ec7357-f2gw6"
Jul 29 04:59:36.995935 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:36.995789    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ec7357-f2gw6" event={"ID":"396e551e-00cf-4358-8f7f-1f2d56fd99a8","Type":"ContainerDied","Data":"79c55dd1cb27812739d30fb6a40791cfe44a6f000a20bfea41facd00043993a2"}
Jul 29 04:59:36.995935 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:36.995859    2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79c55dd1cb27812739d30fb6a40791cfe44a6f000a20bfea41facd00043993a2"
Jul 29 04:59:49.202281 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:49.202247    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ec7357-7d7bcdf6b-fxs5l"]
Jul 29 04:59:49.202930 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:49.202493    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ec7357-7d7bcdf6b-fxs5l" podUID="5a283dca-805f-49b3-93ec-310c2ec07c1d" containerName="maas-api" containerID="cri-o://00be6115785e799e9710e60ccde50e3d513672ead690af85f713ce9160a5e779" gracePeriod=30
Jul 29 04:59:49.253511 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:49.253478    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ec7357-67c6b5c4f4-l5nrc"]
Jul 29 04:59:49.253739 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:49.253713    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-ec7357-67c6b5c4f4-l5nrc" podUID="4b11c4ff-64ad-44bc-9ea2-df83a32ae814" containerName="payload-processing" containerID="cri-o://d9ee6f93250669528ebee75190eafa1009cacb857408df1f2c3aa7add5cb01e6" gracePeriod=30
Jul 29 04:59:49.260072 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:49.260042    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ec7357-69f47849d8-kqpvh"]
Jul 29 04:59:49.260321 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:49.260271    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec7357-69f47849d8-kqpvh" podUID="ea001ef5-473d-4e0b-9437-eabb4094553d" containerName="payload-pre-processing" containerID="cri-o://e1e4c00c6917329c94e4165a602e43fafbb36188784b7f7d135a0b0a9c46cc58" gracePeriod=30
Jul 29 04:59:49.393394 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:49.393356    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ec7357-f2gw6"]
Jul 29 04:59:49.396742 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:49.396711    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ec7357-f2gw6"]
Jul 29 04:59:49.532172 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:49.532141    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ec7357-7d7bcdf6b-fxs5l"
Jul 29 04:59:49.568017 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:49.567990    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ec7357-67c6b5c4f4-l5nrc"
Jul 29 04:59:49.571654 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:49.571632    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec7357-69f47849d8-kqpvh"
Jul 29 04:59:49.650010 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:49.649980    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a283dca-805f-49b3-93ec-310c2ec07c1d-openshift-service-ca\") pod \"5a283dca-805f-49b3-93ec-310c2ec07c1d\" (UID: \"5a283dca-805f-49b3-93ec-310c2ec07c1d\") "
Jul 29 04:59:49.650010 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:49.650010    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a283dca-805f-49b3-93ec-310c2ec07c1d-maas-api-tls\") pod \"5a283dca-805f-49b3-93ec-310c2ec07c1d\" (UID: \"5a283dca-805f-49b3-93ec-310c2ec07c1d\") "
Jul 29 04:59:49.650255 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:49.650045    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b11c4ff-64ad-44bc-9ea2-df83a32ae814-plugins-config-volume\") pod \"4b11c4ff-64ad-44bc-9ea2-df83a32ae814\" (UID: \"4b11c4ff-64ad-44bc-9ea2-df83a32ae814\") "
Jul 29 04:59:49.650255 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:49.650061    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4hmqg\" (UniqueName: \"kubernetes.io/projected/4b11c4ff-64ad-44bc-9ea2-df83a32ae814-kube-api-access-4hmqg\") pod \"4b11c4ff-64ad-44bc-9ea2-df83a32ae814\" (UID: \"4b11c4ff-64ad-44bc-9ea2-df83a32ae814\") "
Jul 29 04:59:49.650255 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:49.650217    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-49f2r\" (UniqueName: \"kubernetes.io/projected/5a283dca-805f-49b3-93ec-310c2ec07c1d-kube-api-access-49f2r\") pod \"5a283dca-805f-49b3-93ec-310c2ec07c1d\" (UID: \"5a283dca-805f-49b3-93ec-310c2ec07c1d\") "
Jul 29 04:59:49.650422 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:49.650259    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jw2fz\" (UniqueName: \"kubernetes.io/projected/ea001ef5-473d-4e0b-9437-eabb4094553d-kube-api-access-jw2fz\") pod \"ea001ef5-473d-4e0b-9437-eabb4094553d\" (UID: \"ea001ef5-473d-4e0b-9437-eabb4094553d\") "
Jul 29 04:59:49.650422 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:49.650375    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea001ef5-473d-4e0b-9437-eabb4094553d-plugins-config-volume\") pod \"ea001ef5-473d-4e0b-9437-eabb4094553d\" (UID: \"ea001ef5-473d-4e0b-9437-eabb4094553d\") "
Jul 29 04:59:49.650422 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:49.650402    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a283dca-805f-49b3-93ec-310c2ec07c1d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5a283dca-805f-49b3-93ec-310c2ec07c1d" (UID: "5a283dca-805f-49b3-93ec-310c2ec07c1d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 04:59:49.650422 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:49.650412    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b11c4ff-64ad-44bc-9ea2-df83a32ae814-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4b11c4ff-64ad-44bc-9ea2-df83a32ae814" (UID: "4b11c4ff-64ad-44bc-9ea2-df83a32ae814"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 04:59:49.650886 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:49.650770    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea001ef5-473d-4e0b-9437-eabb4094553d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ea001ef5-473d-4e0b-9437-eabb4094553d" (UID: "ea001ef5-473d-4e0b-9437-eabb4094553d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 04:59:49.650975 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:49.650909    2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a283dca-805f-49b3-93ec-310c2ec07c1d-openshift-service-ca\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:59:49.650975 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:49.650930    2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b11c4ff-64ad-44bc-9ea2-df83a32ae814-plugins-config-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:59:49.650975 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:49.650943    2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea001ef5-473d-4e0b-9437-eabb4094553d-plugins-config-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:59:49.652342 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:49.652316    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a283dca-805f-49b3-93ec-310c2ec07c1d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5a283dca-805f-49b3-93ec-310c2ec07c1d" (UID: "5a283dca-805f-49b3-93ec-310c2ec07c1d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 04:59:49.652430 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:49.652395    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b11c4ff-64ad-44bc-9ea2-df83a32ae814-kube-api-access-4hmqg" (OuterVolumeSpecName: "kube-api-access-4hmqg") pod "4b11c4ff-64ad-44bc-9ea2-df83a32ae814" (UID: "4b11c4ff-64ad-44bc-9ea2-df83a32ae814"). InnerVolumeSpecName "kube-api-access-4hmqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 04:59:49.652503 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:49.652483    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a283dca-805f-49b3-93ec-310c2ec07c1d-kube-api-access-49f2r" (OuterVolumeSpecName: "kube-api-access-49f2r") pod "5a283dca-805f-49b3-93ec-310c2ec07c1d" (UID: "5a283dca-805f-49b3-93ec-310c2ec07c1d"). InnerVolumeSpecName "kube-api-access-49f2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 04:59:49.652715 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:49.652691    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea001ef5-473d-4e0b-9437-eabb4094553d-kube-api-access-jw2fz" (OuterVolumeSpecName: "kube-api-access-jw2fz") pod "ea001ef5-473d-4e0b-9437-eabb4094553d" (UID: "ea001ef5-473d-4e0b-9437-eabb4094553d"). InnerVolumeSpecName "kube-api-access-jw2fz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 04:59:49.752433 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:49.752354    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49f2r\" (UniqueName: \"kubernetes.io/projected/5a283dca-805f-49b3-93ec-310c2ec07c1d-kube-api-access-49f2r\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:59:49.752433 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:49.752382    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jw2fz\" (UniqueName: \"kubernetes.io/projected/ea001ef5-473d-4e0b-9437-eabb4094553d-kube-api-access-jw2fz\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:59:49.752433 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:49.752395    2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a283dca-805f-49b3-93ec-310c2ec07c1d-maas-api-tls\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:59:49.752433 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:49.752406    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hmqg\" (UniqueName: \"kubernetes.io/projected/4b11c4ff-64ad-44bc-9ea2-df83a32ae814-kube-api-access-4hmqg\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 04:59:49.835387 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:49.835360    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="396e551e-00cf-4358-8f7f-1f2d56fd99a8" path="/var/lib/kubelet/pods/396e551e-00cf-4358-8f7f-1f2d56fd99a8/volumes"
Jul 29 04:59:50.069844 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:50.069724    2570 generic.go:356] "Generic (PLEG): container finished" podID="ea001ef5-473d-4e0b-9437-eabb4094553d" containerID="e1e4c00c6917329c94e4165a602e43fafbb36188784b7f7d135a0b0a9c46cc58" exitCode=0
Jul 29 04:59:50.069844 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:50.069779    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec7357-69f47849d8-kqpvh"
Jul 29 04:59:50.069844 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:50.069810    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec7357-69f47849d8-kqpvh" event={"ID":"ea001ef5-473d-4e0b-9437-eabb4094553d","Type":"ContainerDied","Data":"e1e4c00c6917329c94e4165a602e43fafbb36188784b7f7d135a0b0a9c46cc58"}
Jul 29 04:59:50.070132 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:50.069878    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec7357-69f47849d8-kqpvh" event={"ID":"ea001ef5-473d-4e0b-9437-eabb4094553d","Type":"ContainerDied","Data":"9bcc22d5a7c90f17a56c615656e0e698810821deb478ba2f57bffa5f4fa278bc"}
Jul 29 04:59:50.070132 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:50.069900    2570 scope.go:117] "RemoveContainer" containerID="e1e4c00c6917329c94e4165a602e43fafbb36188784b7f7d135a0b0a9c46cc58"
Jul 29 04:59:50.071159 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:50.071135    2570 generic.go:356] "Generic (PLEG): container finished" podID="4b11c4ff-64ad-44bc-9ea2-df83a32ae814" containerID="d9ee6f93250669528ebee75190eafa1009cacb857408df1f2c3aa7add5cb01e6" exitCode=0
Jul 29 04:59:50.071274 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:50.071177    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ec7357-67c6b5c4f4-l5nrc"
Jul 29 04:59:50.071274 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:50.071225    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ec7357-67c6b5c4f4-l5nrc" event={"ID":"4b11c4ff-64ad-44bc-9ea2-df83a32ae814","Type":"ContainerDied","Data":"d9ee6f93250669528ebee75190eafa1009cacb857408df1f2c3aa7add5cb01e6"}
Jul 29 04:59:50.071523 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:50.071482    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ec7357-67c6b5c4f4-l5nrc" event={"ID":"4b11c4ff-64ad-44bc-9ea2-df83a32ae814","Type":"ContainerDied","Data":"694f0201ff85f4af1b3331dac9aa5192ddaeb494758807f33121f49a6232eb43"}
Jul 29 04:59:50.072699 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:50.072655    2570 generic.go:356] "Generic (PLEG): container finished" podID="5a283dca-805f-49b3-93ec-310c2ec07c1d" containerID="00be6115785e799e9710e60ccde50e3d513672ead690af85f713ce9160a5e779" exitCode=0
Jul 29 04:59:50.072699 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:50.072683    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ec7357-7d7bcdf6b-fxs5l" event={"ID":"5a283dca-805f-49b3-93ec-310c2ec07c1d","Type":"ContainerDied","Data":"00be6115785e799e9710e60ccde50e3d513672ead690af85f713ce9160a5e779"}
Jul 29 04:59:50.072894 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:50.072734    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ec7357-7d7bcdf6b-fxs5l" event={"ID":"5a283dca-805f-49b3-93ec-310c2ec07c1d","Type":"ContainerDied","Data":"5201c14472d43af7ffbc14383927ba44e1e2b79e8626de45b241fcecda86a18a"}
Jul 29 04:59:50.072894 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:50.072692    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ec7357-7d7bcdf6b-fxs5l"
Jul 29 04:59:50.079727 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:50.079713    2570 scope.go:117] "RemoveContainer" containerID="e1e4c00c6917329c94e4165a602e43fafbb36188784b7f7d135a0b0a9c46cc58"
Jul 29 04:59:50.079982 ip-10-0-135-138 kubenswrapper[2570]: E0729 04:59:50.079965    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1e4c00c6917329c94e4165a602e43fafbb36188784b7f7d135a0b0a9c46cc58\": container with ID starting with e1e4c00c6917329c94e4165a602e43fafbb36188784b7f7d135a0b0a9c46cc58 not found: ID does not exist" containerID="e1e4c00c6917329c94e4165a602e43fafbb36188784b7f7d135a0b0a9c46cc58"
Jul 29 04:59:50.080034 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:50.079989    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1e4c00c6917329c94e4165a602e43fafbb36188784b7f7d135a0b0a9c46cc58"} err="failed to get container status \"e1e4c00c6917329c94e4165a602e43fafbb36188784b7f7d135a0b0a9c46cc58\": rpc error: code = NotFound desc = could not find container \"e1e4c00c6917329c94e4165a602e43fafbb36188784b7f7d135a0b0a9c46cc58\": container with ID starting with e1e4c00c6917329c94e4165a602e43fafbb36188784b7f7d135a0b0a9c46cc58 not found: ID does not exist"
Jul 29 04:59:50.080034 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:50.080009    2570 scope.go:117] "RemoveContainer" containerID="d9ee6f93250669528ebee75190eafa1009cacb857408df1f2c3aa7add5cb01e6"
Jul 29 04:59:50.087799 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:50.087782    2570 scope.go:117] "RemoveContainer" containerID="d9ee6f93250669528ebee75190eafa1009cacb857408df1f2c3aa7add5cb01e6"
Jul 29 04:59:50.088163 ip-10-0-135-138 kubenswrapper[2570]: E0729 04:59:50.088144    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9ee6f93250669528ebee75190eafa1009cacb857408df1f2c3aa7add5cb01e6\": container with ID starting with d9ee6f93250669528ebee75190eafa1009cacb857408df1f2c3aa7add5cb01e6 not found: ID does not exist" containerID="d9ee6f93250669528ebee75190eafa1009cacb857408df1f2c3aa7add5cb01e6"
Jul 29 04:59:50.088229 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:50.088170    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9ee6f93250669528ebee75190eafa1009cacb857408df1f2c3aa7add5cb01e6"} err="failed to get container status \"d9ee6f93250669528ebee75190eafa1009cacb857408df1f2c3aa7add5cb01e6\": rpc error: code = NotFound desc = could not find container \"d9ee6f93250669528ebee75190eafa1009cacb857408df1f2c3aa7add5cb01e6\": container with ID starting with d9ee6f93250669528ebee75190eafa1009cacb857408df1f2c3aa7add5cb01e6 not found: ID does not exist"
Jul 29 04:59:50.088229 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:50.088187    2570 scope.go:117] "RemoveContainer" containerID="00be6115785e799e9710e60ccde50e3d513672ead690af85f713ce9160a5e779"
Jul 29 04:59:50.091783 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:50.091763    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ec7357-7d7bcdf6b-fxs5l"]
Jul 29 04:59:50.095350 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:50.095330    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ec7357-7d7bcdf6b-fxs5l"]
Jul 29 04:59:50.096851 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:50.096812    2570 scope.go:117] "RemoveContainer" containerID="00be6115785e799e9710e60ccde50e3d513672ead690af85f713ce9160a5e779"
Jul 29 04:59:50.097136 ip-10-0-135-138 kubenswrapper[2570]: E0729 04:59:50.097118    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00be6115785e799e9710e60ccde50e3d513672ead690af85f713ce9160a5e779\": container with ID starting with 00be6115785e799e9710e60ccde50e3d513672ead690af85f713ce9160a5e779 not found: ID does not exist" containerID="00be6115785e799e9710e60ccde50e3d513672ead690af85f713ce9160a5e779"
Jul 29 04:59:50.097207 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:50.097142    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00be6115785e799e9710e60ccde50e3d513672ead690af85f713ce9160a5e779"} err="failed to get container status \"00be6115785e799e9710e60ccde50e3d513672ead690af85f713ce9160a5e779\": rpc error: code = NotFound desc = could not find container \"00be6115785e799e9710e60ccde50e3d513672ead690af85f713ce9160a5e779\": container with ID starting with 00be6115785e799e9710e60ccde50e3d513672ead690af85f713ce9160a5e779 not found: ID does not exist"
Jul 29 04:59:50.104917 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:50.104897    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ec7357-69f47849d8-kqpvh"]
Jul 29 04:59:50.108460 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:50.108442    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ec7357-69f47849d8-kqpvh"]
Jul 29 04:59:50.118556 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:50.118537    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ec7357-67c6b5c4f4-l5nrc"]
Jul 29 04:59:50.123663 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:50.123640    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ec7357-67c6b5c4f4-l5nrc"]
Jul 29 04:59:51.835490 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:51.835456    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b11c4ff-64ad-44bc-9ea2-df83a32ae814" path="/var/lib/kubelet/pods/4b11c4ff-64ad-44bc-9ea2-df83a32ae814/volumes"
Jul 29 04:59:51.835885 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:51.835874    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a283dca-805f-49b3-93ec-310c2ec07c1d" path="/var/lib/kubelet/pods/5a283dca-805f-49b3-93ec-310c2ec07c1d/volumes"
Jul 29 04:59:51.836190 ip-10-0-135-138 kubenswrapper[2570]: I0729 04:59:51.836176    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea001ef5-473d-4e0b-9437-eabb4094553d" path="/var/lib/kubelet/pods/ea001ef5-473d-4e0b-9437-eabb4094553d/volumes"
Jul 29 05:00:00.129614 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:00:00.129507    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2cb3bf-29755020-bt8rm"]
Jul 29 05:00:00.130222 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:00:00.130150    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b11c4ff-64ad-44bc-9ea2-df83a32ae814" containerName="payload-processing"
Jul 29 05:00:00.130222 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:00:00.130165    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b11c4ff-64ad-44bc-9ea2-df83a32ae814" containerName="payload-processing"
Jul 29 05:00:00.130222 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:00:00.130174    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="396e551e-00cf-4358-8f7f-1f2d56fd99a8" containerName="revoke-keys"
Jul 29 05:00:00.130222 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:00:00.130189    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="396e551e-00cf-4358-8f7f-1f2d56fd99a8" containerName="revoke-keys"
Jul 29 05:00:00.130222 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:00:00.130201    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a283dca-805f-49b3-93ec-310c2ec07c1d" containerName="maas-api"
Jul 29 05:00:00.130222 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:00:00.130207    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a283dca-805f-49b3-93ec-310c2ec07c1d" containerName="maas-api"
Jul 29 05:00:00.130222 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:00:00.130215    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea001ef5-473d-4e0b-9437-eabb4094553d" containerName="payload-pre-processing"
Jul 29 05:00:00.130222 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:00:00.130220    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea001ef5-473d-4e0b-9437-eabb4094553d" containerName="payload-pre-processing"
Jul 29 05:00:00.130504 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:00:00.130294    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b11c4ff-64ad-44bc-9ea2-df83a32ae814" containerName="payload-processing"
Jul 29 05:00:00.130504 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:00:00.130303    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a283dca-805f-49b3-93ec-310c2ec07c1d" containerName="maas-api"
Jul 29 05:00:00.130504 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:00:00.130307    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="396e551e-00cf-4358-8f7f-1f2d56fd99a8" containerName="revoke-keys"
Jul 29 05:00:00.130504 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:00:00.130316    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea001ef5-473d-4e0b-9437-eabb4094553d" containerName="payload-pre-processing"
Jul 29 05:00:00.134340 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:00:00.134320    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755020-dxr62"]
Jul 29 05:00:00.134456 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:00:00.134439    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2cb3bf-29755020-bt8rm"
Jul 29 05:00:00.136926 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:00:00.136906    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8frm\""
Jul 29 05:00:00.137885 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:00:00.137866    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-b550f6-29755020-kb668"]
Jul 29 05:00:00.138045 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:00:00.138018    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755020-dxr62"
Jul 29 05:00:00.141886 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:00:00.141862    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b70f7c-29755020-5hxmm"]
Jul 29 05:00:00.142041 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:00:00.142023    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-b550f6-29755020-kb668"
Jul 29 05:00:00.145915 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:00:00.145891    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2cb3bf-29755020-bt8rm"]
Jul 29 05:00:00.146024 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:00:00.145933    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755020-dxr62"]
Jul 29 05:00:00.146024 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:00:00.146014    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b70f7c-29755020-5hxmm"
Jul 29 05:00:00.146160 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:00:00.146098    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b70f7c-29755020-5hxmm"]
Jul 29 05:00:00.151094 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:00:00.151074    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2cb3bf-29755020-bt8rm"
Jul 29 05:00:00.160697 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:00:00.160366    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755020-dxr62"
Jul 29 05:00:00.160697 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:00:00.160482    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-b550f6-29755020-kb668"]
Jul 29 05:00:00.203403 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:00:00.203363    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-b550f6-29755020-kb668"
Jul 29 05:00:00.211566 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:00:00.211389    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b70f7c-29755020-5hxmm"
Jul 29 05:00:00.349095 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:00:00.349011    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2cb3bf-29755020-bt8rm"]
Jul 29 05:00:00.352597 ip-10-0-135-138 kubenswrapper[2570]: W0729 05:00:00.352567    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1236f357_0732_4ddf_b029_e6fb242df648.slice/crio-1a0d6e93285683374b55083582b8c7345dcad1522d6ba545dc402c69f9aacecb WatchSource:0}: Error finding container 1a0d6e93285683374b55083582b8c7345dcad1522d6ba545dc402c69f9aacecb: Status 404 returned error can't find the container with id 1a0d6e93285683374b55083582b8c7345dcad1522d6ba545dc402c69f9aacecb
Jul 29 05:00:00.382345 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:00:00.382270    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755020-dxr62"]
Jul 29 05:00:00.393608 ip-10-0-135-138 kubenswrapper[2570]: W0729 05:00:00.393568    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dbc9aec_d52f_425d_b506_04eefcd2dcc0.slice/crio-97150d379f0e7abb4d55644ff54ec4a981ce4bbc5a9e533ad1c99e5327856ae2 WatchSource:0}: Error finding container 97150d379f0e7abb4d55644ff54ec4a981ce4bbc5a9e533ad1c99e5327856ae2: Status 404 returned error can't find the container with id 97150d379f0e7abb4d55644ff54ec4a981ce4bbc5a9e533ad1c99e5327856ae2
Jul 29 05:00:00.418232 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:00:00.418205    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-b550f6-29755020-kb668"]
Jul 29 05:00:00.418851 ip-10-0-135-138 kubenswrapper[2570]: W0729 05:00:00.418791    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod862757c8_f510_426e_96be_61657c958aff.slice/crio-682d9df5e2db3693a7d0efc5bc96c0de612793d578e11bd06dc0fb45fc0c758b WatchSource:0}: Error finding container 682d9df5e2db3693a7d0efc5bc96c0de612793d578e11bd06dc0fb45fc0c758b: Status 404 returned error can't find the container with id 682d9df5e2db3693a7d0efc5bc96c0de612793d578e11bd06dc0fb45fc0c758b
Jul 29 05:00:00.441402 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:00:00.441374    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b70f7c-29755020-5hxmm"]
Jul 29 05:00:00.442553 ip-10-0-135-138 kubenswrapper[2570]: W0729 05:00:00.442532    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod316bd834_60d7_4e4b_a97e_e1e6e645d929.slice/crio-269a651a8440b9a276929d9f4ca8f4f8a351441bc4ee7d8953a2d600eacfe6e9 WatchSource:0}: Error finding container 269a651a8440b9a276929d9f4ca8f4f8a351441bc4ee7d8953a2d600eacfe6e9: Status 404 returned error can't find the container with id 269a651a8440b9a276929d9f4ca8f4f8a351441bc4ee7d8953a2d600eacfe6e9
Jul 29 05:00:01.136531 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:00:01.136423    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2cb3bf-29755020-bt8rm" event={"ID":"1236f357-0732-4ddf-b029-e6fb242df648","Type":"ContainerStarted","Data":"5931d8046730cbe7e8269f7d6ef81fafd1605397c4c669278f04dc9e1145b7cf"}
Jul 29 05:00:01.137061 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:00:01.136542    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2cb3bf-29755020-bt8rm" event={"ID":"1236f357-0732-4ddf-b029-e6fb242df648","Type":"ContainerStarted","Data":"1a0d6e93285683374b55083582b8c7345dcad1522d6ba545dc402c69f9aacecb"}
Jul 29 05:00:01.138157 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:00:01.138132    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b70f7c-29755020-5hxmm" event={"ID":"316bd834-60d7-4e4b-a97e-e1e6e645d929","Type":"ContainerStarted","Data":"5df459f99a1733f74e2d966b334bfb8b24c82bf38017be7c873947a0bd0062b4"}
Jul 29 05:00:01.138275 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:00:01.138177    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b70f7c-29755020-5hxmm" event={"ID":"316bd834-60d7-4e4b-a97e-e1e6e645d929","Type":"ContainerStarted","Data":"269a651a8440b9a276929d9f4ca8f4f8a351441bc4ee7d8953a2d600eacfe6e9"}
Jul 29 05:00:01.139720 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:00:01.139689    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755020-dxr62" event={"ID":"8dbc9aec-d52f-425d-b506-04eefcd2dcc0","Type":"ContainerStarted","Data":"b10d220d9e168b37b08f8236ff26fc9745da9bbd2b76059f1e9d4933c179d203"}
Jul 29 05:00:01.139892 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:00:01.139733    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755020-dxr62" event={"ID":"8dbc9aec-d52f-425d-b506-04eefcd2dcc0","Type":"ContainerStarted","Data":"97150d379f0e7abb4d55644ff54ec4a981ce4bbc5a9e533ad1c99e5327856ae2"}
Jul 29 05:00:01.141100 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:00:01.141075    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-b550f6-29755020-kb668" event={"ID":"862757c8-f510-426e-96be-61657c958aff","Type":"ContainerStarted","Data":"6bbd61d389b6b80a9856f1b27d7d71395bfed64f57da5e2209bb8d93e2944211"}
Jul 29 05:00:01.141192 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:00:01.141121    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-b550f6-29755020-kb668" event={"ID":"862757c8-f510-426e-96be-61657c958aff","Type":"ContainerStarted","Data":"682d9df5e2db3693a7d0efc5bc96c0de612793d578e11bd06dc0fb45fc0c758b"}
Jul 29 05:00:01.152699 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:00:01.152652    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2cb3bf-29755020-bt8rm" podStartSLOduration=1.152637002 podStartE2EDuration="1.152637002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 05:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 05:00:01.150976422 +0000 UTC m=+3865.881107491" watchObservedRunningTime="2026-07-29 05:00:01.152637002 +0000 UTC m=+3865.882768071"
Jul 29 05:00:01.170351 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:00:01.170306    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-b550f6-29755020-kb668" podStartSLOduration=1.170292964 podStartE2EDuration="1.170292964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 05:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 05:00:01.16931861 +0000 UTC m=+3865.899449681" watchObservedRunningTime="2026-07-29 05:00:01.170292964 +0000 UTC m=+3865.900424088"
Jul 29 05:00:01.185049 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:00:01.185001    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755020-dxr62" podStartSLOduration=1.184986788 podStartE2EDuration="1.184986788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 05:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 05:00:01.183025337 +0000 UTC m=+3865.913156426" watchObservedRunningTime="2026-07-29 05:00:01.184986788 +0000 UTC m=+3865.915117857"
Jul 29 05:00:01.196507 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:00:01.196469    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b70f7c-29755020-5hxmm" podStartSLOduration=1.196457661 podStartE2EDuration="1.196457661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 05:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 05:00:01.196102166 +0000 UTC m=+3865.926233258" watchObservedRunningTime="2026-07-29 05:00:01.196457661 +0000 UTC m=+3865.926588730"
Jul 29 05:00:02.147653 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:00:02.147620    2570 generic.go:356] "Generic (PLEG): container finished" podID="8dbc9aec-d52f-425d-b506-04eefcd2dcc0" containerID="b10d220d9e168b37b08f8236ff26fc9745da9bbd2b76059f1e9d4933c179d203" exitCode=0
Jul 29 05:00:02.148121 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:00:02.147701    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755020-dxr62" event={"ID":"8dbc9aec-d52f-425d-b506-04eefcd2dcc0","Type":"ContainerDied","Data":"b10d220d9e168b37b08f8236ff26fc9745da9bbd2b76059f1e9d4933c179d203"}
Jul 29 05:00:02.149425 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:00:02.149399    2570 generic.go:356] "Generic (PLEG): container finished" podID="862757c8-f510-426e-96be-61657c958aff" containerID="6bbd61d389b6b80a9856f1b27d7d71395bfed64f57da5e2209bb8d93e2944211" exitCode=0
Jul 29 05:00:02.149555 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:00:02.149473    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-b550f6-29755020-kb668" event={"ID":"862757c8-f510-426e-96be-61657c958aff","Type":"ContainerDied","Data":"6bbd61d389b6b80a9856f1b27d7d71395bfed64f57da5e2209bb8d93e2944211"}
Jul 29 05:00:02.151038 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:00:02.151018    2570 generic.go:356] "Generic (PLEG): container finished" podID="1236f357-0732-4ddf-b029-e6fb242df648" containerID="5931d8046730cbe7e8269f7d6ef81fafd1605397c4c669278f04dc9e1145b7cf" exitCode=0
Jul 29 05:00:02.151132 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:00:02.151104    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2cb3bf-29755020-bt8rm" event={"ID":"1236f357-0732-4ddf-b029-e6fb242df648","Type":"ContainerDied","Data":"5931d8046730cbe7e8269f7d6ef81fafd1605397c4c669278f04dc9e1145b7cf"}
Jul 29 05:00:02.152443 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:00:02.152422    2570 generic.go:356] "Generic (PLEG): container finished" podID="316bd834-60d7-4e4b-a97e-e1e6e645d929" containerID="5df459f99a1733f74e2d966b334bfb8b24c82bf38017be7c873947a0bd0062b4" exitCode=0
Jul 29 05:00:02.152522 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:00:02.152454    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b70f7c-29755020-5hxmm" event={"ID":"316bd834-60d7-4e4b-a97e-e1e6e645d929","Type":"ContainerDied","Data":"5df459f99a1733f74e2d966b334bfb8b24c82bf38017be7c873947a0bd0062b4"}
Jul 29 05:00:03.336010 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:00:03.335983    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b70f7c-29755020-5hxmm"
Jul 29 05:00:03.384961 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:00:03.384895    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755020-dxr62"
Jul 29 05:00:03.388435 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:00:03.388383    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-b550f6-29755020-kb668"
Jul 29 05:00:03.391866 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:00:03.391847    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2cb3bf-29755020-bt8rm"
Jul 29 05:00:04.163330 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:00:04.163257    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2cb3bf-29755020-bt8rm"
Jul 29 05:00:04.163480 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:00:04.163256    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2cb3bf-29755020-bt8rm" event={"ID":"1236f357-0732-4ddf-b029-e6fb242df648","Type":"ContainerDied","Data":"1a0d6e93285683374b55083582b8c7345dcad1522d6ba545dc402c69f9aacecb"}
Jul 29 05:00:04.163480 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:00:04.163371    2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a0d6e93285683374b55083582b8c7345dcad1522d6ba545dc402c69f9aacecb"
Jul 29 05:00:04.164544 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:00:04.164525    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b70f7c-29755020-5hxmm"
Jul 29 05:00:04.164675 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:00:04.164552    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b70f7c-29755020-5hxmm" event={"ID":"316bd834-60d7-4e4b-a97e-e1e6e645d929","Type":"ContainerDied","Data":"269a651a8440b9a276929d9f4ca8f4f8a351441bc4ee7d8953a2d600eacfe6e9"}
Jul 29 05:00:04.164675 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:00:04.164594    2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="269a651a8440b9a276929d9f4ca8f4f8a351441bc4ee7d8953a2d600eacfe6e9"
Jul 29 05:00:04.165869 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:00:04.165854    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755020-dxr62"
Jul 29 05:00:04.165964 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:00:04.165894    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755020-dxr62" event={"ID":"8dbc9aec-d52f-425d-b506-04eefcd2dcc0","Type":"ContainerDied","Data":"97150d379f0e7abb4d55644ff54ec4a981ce4bbc5a9e533ad1c99e5327856ae2"}
Jul 29 05:00:04.165964 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:00:04.165938    2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97150d379f0e7abb4d55644ff54ec4a981ce4bbc5a9e533ad1c99e5327856ae2"
Jul 29 05:00:04.167222 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:00:04.167200    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-b550f6-29755020-kb668" event={"ID":"862757c8-f510-426e-96be-61657c958aff","Type":"ContainerDied","Data":"682d9df5e2db3693a7d0efc5bc96c0de612793d578e11bd06dc0fb45fc0c758b"}
Jul 29 05:00:04.167289 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:00:04.167243    2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="682d9df5e2db3693a7d0efc5bc96c0de612793d578e11bd06dc0fb45fc0c758b"
Jul 29 05:00:04.167289 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:00:04.167209    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-b550f6-29755020-kb668"
Jul 29 05:01:02.106358 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:02.106326    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2cb3bf-77cf9bcfff-k67d8"]
Jul 29 05:01:02.128295 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:02.128265    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2cb3bf-29755020-bt8rm"]
Jul 29 05:01:02.130687 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:02.130663    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2cb3bf-29755020-bt8rm"]
Jul 29 05:01:02.163309 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:02.163278    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2cb3bf-6749775684-rg6nk"]
Jul 29 05:01:02.163631 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:02.163580    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2cb3bf-6749775684-rg6nk" podUID="77cdd15d-361a-494b-9295-ba5f3810f12c" containerName="payload-pre-processing" containerID="cri-o://ca7c4450c054bf7622fb662daa7c2632ce72b3c2728e010215cc5d222576f914" gracePeriod=30
Jul 29 05:01:02.440495 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:02.440466    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2cb3bf-6749775684-rg6nk"
Jul 29 05:01:02.483908 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:02.483874    2570 generic.go:356] "Generic (PLEG): container finished" podID="77cdd15d-361a-494b-9295-ba5f3810f12c" containerID="ca7c4450c054bf7622fb662daa7c2632ce72b3c2728e010215cc5d222576f914" exitCode=0
Jul 29 05:01:02.484101 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:02.483919    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2cb3bf-6749775684-rg6nk"
Jul 29 05:01:02.484101 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:02.483940    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2cb3bf-6749775684-rg6nk" event={"ID":"77cdd15d-361a-494b-9295-ba5f3810f12c","Type":"ContainerDied","Data":"ca7c4450c054bf7622fb662daa7c2632ce72b3c2728e010215cc5d222576f914"}
Jul 29 05:01:02.484101 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:02.483999    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2cb3bf-6749775684-rg6nk" event={"ID":"77cdd15d-361a-494b-9295-ba5f3810f12c","Type":"ContainerDied","Data":"72b50809fdc5d5cf13de7d06a5abefbf36c9a2df0224d5f5abc08b4547070164"}
Jul 29 05:01:02.484101 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:02.484036    2570 scope.go:117] "RemoveContainer" containerID="ca7c4450c054bf7622fb662daa7c2632ce72b3c2728e010215cc5d222576f914"
Jul 29 05:01:02.484348 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:02.484161    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2cb3bf-77cf9bcfff-k67d8" podUID="182fdd57-ce2f-4ca0-acc3-a781eb32b6e5" containerName="maas-api" containerID="cri-o://2d245a8943376cbe54442abc4cfacd6d6231ca8bc274cf145f80782c3d17cc34" gracePeriod=30
Jul 29 05:01:02.494117 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:02.494099    2570 scope.go:117] "RemoveContainer" containerID="ca7c4450c054bf7622fb662daa7c2632ce72b3c2728e010215cc5d222576f914"
Jul 29 05:01:02.494413 ip-10-0-135-138 kubenswrapper[2570]: E0729 05:01:02.494388    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca7c4450c054bf7622fb662daa7c2632ce72b3c2728e010215cc5d222576f914\": container with ID starting with ca7c4450c054bf7622fb662daa7c2632ce72b3c2728e010215cc5d222576f914 not found: ID does not exist" containerID="ca7c4450c054bf7622fb662daa7c2632ce72b3c2728e010215cc5d222576f914"
Jul 29 05:01:02.494521 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:02.494421    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca7c4450c054bf7622fb662daa7c2632ce72b3c2728e010215cc5d222576f914"} err="failed to get container status \"ca7c4450c054bf7622fb662daa7c2632ce72b3c2728e010215cc5d222576f914\": rpc error: code = NotFound desc = could not find container \"ca7c4450c054bf7622fb662daa7c2632ce72b3c2728e010215cc5d222576f914\": container with ID starting with ca7c4450c054bf7622fb662daa7c2632ce72b3c2728e010215cc5d222576f914 not found: ID does not exist"
Jul 29 05:01:02.524254 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:02.524234    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nsfds\" (UniqueName: \"kubernetes.io/projected/77cdd15d-361a-494b-9295-ba5f3810f12c-kube-api-access-nsfds\") pod \"77cdd15d-361a-494b-9295-ba5f3810f12c\" (UID: \"77cdd15d-361a-494b-9295-ba5f3810f12c\") "
Jul 29 05:01:02.524385 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:02.524276    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77cdd15d-361a-494b-9295-ba5f3810f12c-plugins-config-volume\") pod \"77cdd15d-361a-494b-9295-ba5f3810f12c\" (UID: \"77cdd15d-361a-494b-9295-ba5f3810f12c\") "
Jul 29 05:01:02.524652 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:02.524626    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77cdd15d-361a-494b-9295-ba5f3810f12c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "77cdd15d-361a-494b-9295-ba5f3810f12c" (UID: "77cdd15d-361a-494b-9295-ba5f3810f12c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 05:01:02.526413 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:02.526394    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77cdd15d-361a-494b-9295-ba5f3810f12c-kube-api-access-nsfds" (OuterVolumeSpecName: "kube-api-access-nsfds") pod "77cdd15d-361a-494b-9295-ba5f3810f12c" (UID: "77cdd15d-361a-494b-9295-ba5f3810f12c"). InnerVolumeSpecName "kube-api-access-nsfds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 05:01:02.625581 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:02.625539    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsfds\" (UniqueName: \"kubernetes.io/projected/77cdd15d-361a-494b-9295-ba5f3810f12c-kube-api-access-nsfds\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 05:01:02.625581 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:02.625572    2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77cdd15d-361a-494b-9295-ba5f3810f12c-plugins-config-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 05:01:02.727140 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:02.727115    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2cb3bf-77cf9bcfff-k67d8"
Jul 29 05:01:02.809796 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:02.809765    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2cb3bf-6749775684-rg6nk"]
Jul 29 05:01:02.814393 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:02.814364    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2cb3bf-6749775684-rg6nk"]
Jul 29 05:01:02.826658 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:02.826636    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/182fdd57-ce2f-4ca0-acc3-a781eb32b6e5-openshift-service-ca\") pod \"182fdd57-ce2f-4ca0-acc3-a781eb32b6e5\" (UID: \"182fdd57-ce2f-4ca0-acc3-a781eb32b6e5\") "
Jul 29 05:01:02.826774 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:02.826735    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/182fdd57-ce2f-4ca0-acc3-a781eb32b6e5-maas-api-tls\") pod \"182fdd57-ce2f-4ca0-acc3-a781eb32b6e5\" (UID: \"182fdd57-ce2f-4ca0-acc3-a781eb32b6e5\") "
Jul 29 05:01:02.826774 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:02.826755    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pzlkg\" (UniqueName: \"kubernetes.io/projected/182fdd57-ce2f-4ca0-acc3-a781eb32b6e5-kube-api-access-pzlkg\") pod \"182fdd57-ce2f-4ca0-acc3-a781eb32b6e5\" (UID: \"182fdd57-ce2f-4ca0-acc3-a781eb32b6e5\") "
Jul 29 05:01:02.827053 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:02.827029    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/182fdd57-ce2f-4ca0-acc3-a781eb32b6e5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "182fdd57-ce2f-4ca0-acc3-a781eb32b6e5" (UID: "182fdd57-ce2f-4ca0-acc3-a781eb32b6e5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 05:01:02.828963 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:02.828940    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/182fdd57-ce2f-4ca0-acc3-a781eb32b6e5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "182fdd57-ce2f-4ca0-acc3-a781eb32b6e5" (UID: "182fdd57-ce2f-4ca0-acc3-a781eb32b6e5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 05:01:02.829038 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:02.828988    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/182fdd57-ce2f-4ca0-acc3-a781eb32b6e5-kube-api-access-pzlkg" (OuterVolumeSpecName: "kube-api-access-pzlkg") pod "182fdd57-ce2f-4ca0-acc3-a781eb32b6e5" (UID: "182fdd57-ce2f-4ca0-acc3-a781eb32b6e5"). InnerVolumeSpecName "kube-api-access-pzlkg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 05:01:02.927930 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:02.927901    2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/182fdd57-ce2f-4ca0-acc3-a781eb32b6e5-maas-api-tls\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 05:01:02.927930 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:02.927924    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzlkg\" (UniqueName: \"kubernetes.io/projected/182fdd57-ce2f-4ca0-acc3-a781eb32b6e5-kube-api-access-pzlkg\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 05:01:02.927930 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:02.927933    2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/182fdd57-ce2f-4ca0-acc3-a781eb32b6e5-openshift-service-ca\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 05:01:03.490203 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:03.490127    2570 generic.go:356] "Generic (PLEG): container finished" podID="182fdd57-ce2f-4ca0-acc3-a781eb32b6e5" containerID="2d245a8943376cbe54442abc4cfacd6d6231ca8bc274cf145f80782c3d17cc34" exitCode=0
Jul 29 05:01:03.490203 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:03.490173    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2cb3bf-77cf9bcfff-k67d8"
Jul 29 05:01:03.490653 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:03.490221    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2cb3bf-77cf9bcfff-k67d8" event={"ID":"182fdd57-ce2f-4ca0-acc3-a781eb32b6e5","Type":"ContainerDied","Data":"2d245a8943376cbe54442abc4cfacd6d6231ca8bc274cf145f80782c3d17cc34"}
Jul 29 05:01:03.490653 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:03.490273    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2cb3bf-77cf9bcfff-k67d8" event={"ID":"182fdd57-ce2f-4ca0-acc3-a781eb32b6e5","Type":"ContainerDied","Data":"1457fad88ed68c3da13fd3e68c3c2fbda18ecc6877c871993ad5471dfdd03c60"}
Jul 29 05:01:03.490653 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:03.490300    2570 scope.go:117] "RemoveContainer" containerID="2d245a8943376cbe54442abc4cfacd6d6231ca8bc274cf145f80782c3d17cc34"
Jul 29 05:01:03.507382 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:03.507363    2570 scope.go:117] "RemoveContainer" containerID="2d245a8943376cbe54442abc4cfacd6d6231ca8bc274cf145f80782c3d17cc34"
Jul 29 05:01:03.507672 ip-10-0-135-138 kubenswrapper[2570]: E0729 05:01:03.507652    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d245a8943376cbe54442abc4cfacd6d6231ca8bc274cf145f80782c3d17cc34\": container with ID starting with 2d245a8943376cbe54442abc4cfacd6d6231ca8bc274cf145f80782c3d17cc34 not found: ID does not exist" containerID="2d245a8943376cbe54442abc4cfacd6d6231ca8bc274cf145f80782c3d17cc34"
Jul 29 05:01:03.507735 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:03.507685    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d245a8943376cbe54442abc4cfacd6d6231ca8bc274cf145f80782c3d17cc34"} err="failed to get container status \"2d245a8943376cbe54442abc4cfacd6d6231ca8bc274cf145f80782c3d17cc34\": rpc error: code = NotFound desc = could not find container \"2d245a8943376cbe54442abc4cfacd6d6231ca8bc274cf145f80782c3d17cc34\": container with ID starting with 2d245a8943376cbe54442abc4cfacd6d6231ca8bc274cf145f80782c3d17cc34 not found: ID does not exist"
Jul 29 05:01:03.514942 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:03.514918    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2cb3bf-77cf9bcfff-k67d8"]
Jul 29 05:01:03.519986 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:03.519964    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2cb3bf-77cf9bcfff-k67d8"]
Jul 29 05:01:03.835735 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:03.835701    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1236f357-0732-4ddf-b029-e6fb242df648" path="/var/lib/kubelet/pods/1236f357-0732-4ddf-b029-e6fb242df648/volumes"
Jul 29 05:01:03.836072 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:03.836057    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="182fdd57-ce2f-4ca0-acc3-a781eb32b6e5" path="/var/lib/kubelet/pods/182fdd57-ce2f-4ca0-acc3-a781eb32b6e5/volumes"
Jul 29 05:01:03.836390 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:03.836377    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77cdd15d-361a-494b-9295-ba5f3810f12c" path="/var/lib/kubelet/pods/77cdd15d-361a-494b-9295-ba5f3810f12c/volumes"
Jul 29 05:01:19.432674 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:19.432637    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-b70f7c-c9dd4b449-xhff9"]
Jul 29 05:01:19.433307 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:19.432891    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b70f7c-c9dd4b449-xhff9" podUID="69c5c2d6-a4dc-4108-91f5-e1fb2dd7234b" containerName="maas-api" containerID="cri-o://dc8ead7183969f65ea8cc10c5979c158b35eee34dff43df807daae56d2d5a86e" gracePeriod=30
Jul 29 05:01:19.461628 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:19.461602    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b70f7c-29755020-5hxmm"]
Jul 29 05:01:19.463414 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:19.463393    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b70f7c-29755020-5hxmm"]
Jul 29 05:01:19.491722 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:19.491688    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-b70f7c-5f8c686d6b-7c6xx"]
Jul 29 05:01:19.491955 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:19.491928    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-b70f7c-5f8c686d6b-7c6xx" podUID="f719bfad-7662-4706-aed4-c080a4912db2" containerName="payload-processing" containerID="cri-o://ff8556ea9c959699ac11804b8357ca74c6b66ea198af498aa54a76c825bcef30" gracePeriod=30
Jul 29 05:01:19.499670 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:19.499423    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-b70f7c-f59c576c6-5p9q7"]
Jul 29 05:01:19.499670 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:19.499638    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-b70f7c-f59c576c6-5p9q7" podUID="57aa4758-0757-4c92-b9c6-d3eb724a6a37" containerName="payload-pre-processing" containerID="cri-o://62838deb44f595c26ef8ca4d494eb77eab4963c0c671436f28119dffe8d57f01" gracePeriod=30
Jul 29 05:01:19.554711 ip-10-0-135-138 kubenswrapper[2570]: E0729 05:01:19.554677    2570 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf719bfad_7662_4706_aed4_c080a4912db2.slice/crio-ff8556ea9c959699ac11804b8357ca74c6b66ea198af498aa54a76c825bcef30.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57aa4758_0757_4c92_b9c6_d3eb724a6a37.slice/crio-conmon-62838deb44f595c26ef8ca4d494eb77eab4963c0c671436f28119dffe8d57f01.scope\": RecentStats: unable to find data in memory cache]"
Jul 29 05:01:19.577720 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:19.577691    2570 generic.go:356] "Generic (PLEG): container finished" podID="69c5c2d6-a4dc-4108-91f5-e1fb2dd7234b" containerID="dc8ead7183969f65ea8cc10c5979c158b35eee34dff43df807daae56d2d5a86e" exitCode=0
Jul 29 05:01:19.577896 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:19.577757    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b70f7c-c9dd4b449-xhff9" event={"ID":"69c5c2d6-a4dc-4108-91f5-e1fb2dd7234b","Type":"ContainerDied","Data":"dc8ead7183969f65ea8cc10c5979c158b35eee34dff43df807daae56d2d5a86e"}
Jul 29 05:01:19.771496 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:19.771469    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b70f7c-c9dd4b449-xhff9"
Jul 29 05:01:19.774402 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:19.774383    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-b70f7c-f59c576c6-5p9q7"
Jul 29 05:01:19.806347 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:19.806320    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-b70f7c-5f8c686d6b-7c6xx"
Jul 29 05:01:19.835654 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:19.835625    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="316bd834-60d7-4e4b-a97e-e1e6e645d929" path="/var/lib/kubelet/pods/316bd834-60d7-4e4b-a97e-e1e6e645d929/volumes"
Jul 29 05:01:19.892483 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:19.892455    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jzf2n\" (UniqueName: \"kubernetes.io/projected/f719bfad-7662-4706-aed4-c080a4912db2-kube-api-access-jzf2n\") pod \"f719bfad-7662-4706-aed4-c080a4912db2\" (UID: \"f719bfad-7662-4706-aed4-c080a4912db2\") "
Jul 29 05:01:19.892660 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:19.892502    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57aa4758-0757-4c92-b9c6-d3eb724a6a37-plugins-config-volume\") pod \"57aa4758-0757-4c92-b9c6-d3eb724a6a37\" (UID: \"57aa4758-0757-4c92-b9c6-d3eb724a6a37\") "
Jul 29 05:01:19.892660 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:19.892554    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69c5c2d6-a4dc-4108-91f5-e1fb2dd7234b-maas-api-tls\") pod \"69c5c2d6-a4dc-4108-91f5-e1fb2dd7234b\" (UID: \"69c5c2d6-a4dc-4108-91f5-e1fb2dd7234b\") "
Jul 29 05:01:19.892660 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:19.892587    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f719bfad-7662-4706-aed4-c080a4912db2-plugins-config-volume\") pod \"f719bfad-7662-4706-aed4-c080a4912db2\" (UID: \"f719bfad-7662-4706-aed4-c080a4912db2\") "
Jul 29 05:01:19.892660 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:19.892611    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vmmpf\" (UniqueName: \"kubernetes.io/projected/69c5c2d6-a4dc-4108-91f5-e1fb2dd7234b-kube-api-access-vmmpf\") pod \"69c5c2d6-a4dc-4108-91f5-e1fb2dd7234b\" (UID: \"69c5c2d6-a4dc-4108-91f5-e1fb2dd7234b\") "
Jul 29 05:01:19.892660 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:19.892643    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69c5c2d6-a4dc-4108-91f5-e1fb2dd7234b-openshift-service-ca\") pod \"69c5c2d6-a4dc-4108-91f5-e1fb2dd7234b\" (UID: \"69c5c2d6-a4dc-4108-91f5-e1fb2dd7234b\") "
Jul 29 05:01:19.893001 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:19.892665    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4hpft\" (UniqueName: \"kubernetes.io/projected/57aa4758-0757-4c92-b9c6-d3eb724a6a37-kube-api-access-4hpft\") pod \"57aa4758-0757-4c92-b9c6-d3eb724a6a37\" (UID: \"57aa4758-0757-4c92-b9c6-d3eb724a6a37\") "
Jul 29 05:01:19.893001 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:19.892986    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57aa4758-0757-4c92-b9c6-d3eb724a6a37-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "57aa4758-0757-4c92-b9c6-d3eb724a6a37" (UID: "57aa4758-0757-4c92-b9c6-d3eb724a6a37"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 05:01:19.893102 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:19.893041    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69c5c2d6-a4dc-4108-91f5-e1fb2dd7234b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "69c5c2d6-a4dc-4108-91f5-e1fb2dd7234b" (UID: "69c5c2d6-a4dc-4108-91f5-e1fb2dd7234b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 05:01:19.893210 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:19.893179    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f719bfad-7662-4706-aed4-c080a4912db2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f719bfad-7662-4706-aed4-c080a4912db2" (UID: "f719bfad-7662-4706-aed4-c080a4912db2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 05:01:19.894884 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:19.894860    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f719bfad-7662-4706-aed4-c080a4912db2-kube-api-access-jzf2n" (OuterVolumeSpecName: "kube-api-access-jzf2n") pod "f719bfad-7662-4706-aed4-c080a4912db2" (UID: "f719bfad-7662-4706-aed4-c080a4912db2"). InnerVolumeSpecName "kube-api-access-jzf2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 05:01:19.894981 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:19.894900    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69c5c2d6-a4dc-4108-91f5-e1fb2dd7234b-kube-api-access-vmmpf" (OuterVolumeSpecName: "kube-api-access-vmmpf") pod "69c5c2d6-a4dc-4108-91f5-e1fb2dd7234b" (UID: "69c5c2d6-a4dc-4108-91f5-e1fb2dd7234b"). InnerVolumeSpecName "kube-api-access-vmmpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 05:01:19.894981 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:19.894917    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69c5c2d6-a4dc-4108-91f5-e1fb2dd7234b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "69c5c2d6-a4dc-4108-91f5-e1fb2dd7234b" (UID: "69c5c2d6-a4dc-4108-91f5-e1fb2dd7234b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 05:01:19.895227 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:19.895209    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57aa4758-0757-4c92-b9c6-d3eb724a6a37-kube-api-access-4hpft" (OuterVolumeSpecName: "kube-api-access-4hpft") pod "57aa4758-0757-4c92-b9c6-d3eb724a6a37" (UID: "57aa4758-0757-4c92-b9c6-d3eb724a6a37"). InnerVolumeSpecName "kube-api-access-4hpft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 05:01:19.993644 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:19.993559    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzf2n\" (UniqueName: \"kubernetes.io/projected/f719bfad-7662-4706-aed4-c080a4912db2-kube-api-access-jzf2n\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 05:01:19.993644 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:19.993587    2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57aa4758-0757-4c92-b9c6-d3eb724a6a37-plugins-config-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 05:01:19.993644 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:19.993598    2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69c5c2d6-a4dc-4108-91f5-e1fb2dd7234b-maas-api-tls\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 05:01:19.993644 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:19.993607    2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f719bfad-7662-4706-aed4-c080a4912db2-plugins-config-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 05:01:19.993644 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:19.993617    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmmpf\" (UniqueName: \"kubernetes.io/projected/69c5c2d6-a4dc-4108-91f5-e1fb2dd7234b-kube-api-access-vmmpf\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 05:01:19.993644 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:19.993625    2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69c5c2d6-a4dc-4108-91f5-e1fb2dd7234b-openshift-service-ca\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 05:01:19.993644 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:19.993634    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hpft\" (UniqueName: \"kubernetes.io/projected/57aa4758-0757-4c92-b9c6-d3eb724a6a37-kube-api-access-4hpft\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 05:01:20.584507 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:20.584473    2570 generic.go:356] "Generic (PLEG): container finished" podID="f719bfad-7662-4706-aed4-c080a4912db2" containerID="ff8556ea9c959699ac11804b8357ca74c6b66ea198af498aa54a76c825bcef30" exitCode=0
Jul 29 05:01:20.584985 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:20.584515    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-b70f7c-5f8c686d6b-7c6xx"
Jul 29 05:01:20.584985 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:20.584563    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-b70f7c-5f8c686d6b-7c6xx" event={"ID":"f719bfad-7662-4706-aed4-c080a4912db2","Type":"ContainerDied","Data":"ff8556ea9c959699ac11804b8357ca74c6b66ea198af498aa54a76c825bcef30"}
Jul 29 05:01:20.584985 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:20.584623    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-b70f7c-5f8c686d6b-7c6xx" event={"ID":"f719bfad-7662-4706-aed4-c080a4912db2","Type":"ContainerDied","Data":"878d8c64fa34ec0d982617f113dda5d29c2dfeae4564f7dcc92d8ab135eb1b12"}
Jul 29 05:01:20.584985 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:20.584658    2570 scope.go:117] "RemoveContainer" containerID="ff8556ea9c959699ac11804b8357ca74c6b66ea198af498aa54a76c825bcef30"
Jul 29 05:01:20.586014 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:20.585934    2570 generic.go:356] "Generic (PLEG): container finished" podID="57aa4758-0757-4c92-b9c6-d3eb724a6a37" containerID="62838deb44f595c26ef8ca4d494eb77eab4963c0c671436f28119dffe8d57f01" exitCode=0
Jul 29 05:01:20.586014 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:20.585965    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-b70f7c-f59c576c6-5p9q7"
Jul 29 05:01:20.586138 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:20.586012    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-b70f7c-f59c576c6-5p9q7" event={"ID":"57aa4758-0757-4c92-b9c6-d3eb724a6a37","Type":"ContainerDied","Data":"62838deb44f595c26ef8ca4d494eb77eab4963c0c671436f28119dffe8d57f01"}
Jul 29 05:01:20.586138 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:20.586060    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-b70f7c-f59c576c6-5p9q7" event={"ID":"57aa4758-0757-4c92-b9c6-d3eb724a6a37","Type":"ContainerDied","Data":"563f5cd0482a1b69b79d92cfc4c6e668934e0d4bc6d7110b71d86ce6af1c965a"}
Jul 29 05:01:20.587424 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:20.587401    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b70f7c-c9dd4b449-xhff9" event={"ID":"69c5c2d6-a4dc-4108-91f5-e1fb2dd7234b","Type":"ContainerDied","Data":"e11cd7804322ea79f5ef62139455eafb2aaf5857058c9359d9b483f3fa0d0ebc"}
Jul 29 05:01:20.587424 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:20.587420    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b70f7c-c9dd4b449-xhff9"
Jul 29 05:01:20.595206 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:20.595187    2570 scope.go:117] "RemoveContainer" containerID="ff8556ea9c959699ac11804b8357ca74c6b66ea198af498aa54a76c825bcef30"
Jul 29 05:01:20.595452 ip-10-0-135-138 kubenswrapper[2570]: E0729 05:01:20.595428    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff8556ea9c959699ac11804b8357ca74c6b66ea198af498aa54a76c825bcef30\": container with ID starting with ff8556ea9c959699ac11804b8357ca74c6b66ea198af498aa54a76c825bcef30 not found: ID does not exist" containerID="ff8556ea9c959699ac11804b8357ca74c6b66ea198af498aa54a76c825bcef30"
Jul 29 05:01:20.595575 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:20.595453    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff8556ea9c959699ac11804b8357ca74c6b66ea198af498aa54a76c825bcef30"} err="failed to get container status \"ff8556ea9c959699ac11804b8357ca74c6b66ea198af498aa54a76c825bcef30\": rpc error: code = NotFound desc = could not find container \"ff8556ea9c959699ac11804b8357ca74c6b66ea198af498aa54a76c825bcef30\": container with ID starting with ff8556ea9c959699ac11804b8357ca74c6b66ea198af498aa54a76c825bcef30 not found: ID does not exist"
Jul 29 05:01:20.595575 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:20.595473    2570 scope.go:117] "RemoveContainer" containerID="62838deb44f595c26ef8ca4d494eb77eab4963c0c671436f28119dffe8d57f01"
Jul 29 05:01:20.607706 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:20.607621    2570 scope.go:117] "RemoveContainer" containerID="62838deb44f595c26ef8ca4d494eb77eab4963c0c671436f28119dffe8d57f01"
Jul 29 05:01:20.608217 ip-10-0-135-138 kubenswrapper[2570]: E0729 05:01:20.608197    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62838deb44f595c26ef8ca4d494eb77eab4963c0c671436f28119dffe8d57f01\": container with ID starting with 62838deb44f595c26ef8ca4d494eb77eab4963c0c671436f28119dffe8d57f01 not found: ID does not exist" containerID="62838deb44f595c26ef8ca4d494eb77eab4963c0c671436f28119dffe8d57f01"
Jul 29 05:01:20.608321 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:20.608223    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62838deb44f595c26ef8ca4d494eb77eab4963c0c671436f28119dffe8d57f01"} err="failed to get container status \"62838deb44f595c26ef8ca4d494eb77eab4963c0c671436f28119dffe8d57f01\": rpc error: code = NotFound desc = could not find container \"62838deb44f595c26ef8ca4d494eb77eab4963c0c671436f28119dffe8d57f01\": container with ID starting with 62838deb44f595c26ef8ca4d494eb77eab4963c0c671436f28119dffe8d57f01 not found: ID does not exist"
Jul 29 05:01:20.608321 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:20.608245    2570 scope.go:117] "RemoveContainer" containerID="dc8ead7183969f65ea8cc10c5979c158b35eee34dff43df807daae56d2d5a86e"
Jul 29 05:01:20.610485 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:20.610465    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-b70f7c-5f8c686d6b-7c6xx"]
Jul 29 05:01:20.615034 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:20.615017    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-b70f7c-5f8c686d6b-7c6xx"]
Jul 29 05:01:20.624233 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:20.624199    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-b70f7c-c9dd4b449-xhff9"]
Jul 29 05:01:20.626251 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:20.626233    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-b70f7c-c9dd4b449-xhff9"]
Jul 29 05:01:20.637600 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:20.637581    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-b70f7c-f59c576c6-5p9q7"]
Jul 29 05:01:20.649280 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:20.644348    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-b70f7c-f59c576c6-5p9q7"]
Jul 29 05:01:21.835344 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:21.835312    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57aa4758-0757-4c92-b9c6-d3eb724a6a37" path="/var/lib/kubelet/pods/57aa4758-0757-4c92-b9c6-d3eb724a6a37/volumes"
Jul 29 05:01:21.835727 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:21.835701    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69c5c2d6-a4dc-4108-91f5-e1fb2dd7234b" path="/var/lib/kubelet/pods/69c5c2d6-a4dc-4108-91f5-e1fb2dd7234b/volumes"
Jul 29 05:01:21.836065 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:21.836051    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f719bfad-7662-4706-aed4-c080a4912db2" path="/var/lib/kubelet/pods/f719bfad-7662-4706-aed4-c080a4912db2/volumes"
Jul 29 05:01:26.478760 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:26.478685    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b550f6-8kbl2"]
Jul 29 05:01:26.479128 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:26.479076    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69c5c2d6-a4dc-4108-91f5-e1fb2dd7234b" containerName="maas-api"
Jul 29 05:01:26.479128 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:26.479088    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="69c5c2d6-a4dc-4108-91f5-e1fb2dd7234b" containerName="maas-api"
Jul 29 05:01:26.479128 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:26.479096    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f719bfad-7662-4706-aed4-c080a4912db2" containerName="payload-processing"
Jul 29 05:01:26.479128 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:26.479103    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="f719bfad-7662-4706-aed4-c080a4912db2" containerName="payload-processing"
Jul 29 05:01:26.479128 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:26.479115    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57aa4758-0757-4c92-b9c6-d3eb724a6a37" containerName="payload-pre-processing"
Jul 29 05:01:26.479128 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:26.479121    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="57aa4758-0757-4c92-b9c6-d3eb724a6a37" containerName="payload-pre-processing"
Jul 29 05:01:26.479128 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:26.479128    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="862757c8-f510-426e-96be-61657c958aff" containerName="cleanup"
Jul 29 05:01:26.479349 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:26.479133    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="862757c8-f510-426e-96be-61657c958aff" containerName="cleanup"
Jul 29 05:01:26.479349 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:26.479142    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="182fdd57-ce2f-4ca0-acc3-a781eb32b6e5" containerName="maas-api"
Jul 29 05:01:26.479349 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:26.479147    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="182fdd57-ce2f-4ca0-acc3-a781eb32b6e5" containerName="maas-api"
Jul 29 05:01:26.479349 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:26.479152    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8dbc9aec-d52f-425d-b506-04eefcd2dcc0" containerName="cleanup"
Jul 29 05:01:26.479349 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:26.479159    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="8dbc9aec-d52f-425d-b506-04eefcd2dcc0" containerName="cleanup"
Jul 29 05:01:26.479349 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:26.479169    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1236f357-0732-4ddf-b029-e6fb242df648" containerName="cleanup"
Jul 29 05:01:26.479349 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:26.479173    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="1236f357-0732-4ddf-b029-e6fb242df648" containerName="cleanup"
Jul 29 05:01:26.479349 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:26.479179    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="316bd834-60d7-4e4b-a97e-e1e6e645d929" containerName="cleanup"
Jul 29 05:01:26.479349 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:26.479184    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="316bd834-60d7-4e4b-a97e-e1e6e645d929" containerName="cleanup"
Jul 29 05:01:26.479349 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:26.479194    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77cdd15d-361a-494b-9295-ba5f3810f12c" containerName="payload-pre-processing"
Jul 29 05:01:26.479349 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:26.479199    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="77cdd15d-361a-494b-9295-ba5f3810f12c" containerName="payload-pre-processing"
Jul 29 05:01:26.479349 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:26.479254    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="316bd834-60d7-4e4b-a97e-e1e6e645d929" containerName="cleanup"
Jul 29 05:01:26.479349 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:26.479262    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="77cdd15d-361a-494b-9295-ba5f3810f12c" containerName="payload-pre-processing"
Jul 29 05:01:26.479349 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:26.479268    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="862757c8-f510-426e-96be-61657c958aff" containerName="cleanup"
Jul 29 05:01:26.479349 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:26.479275    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="57aa4758-0757-4c92-b9c6-d3eb724a6a37" containerName="payload-pre-processing"
Jul 29 05:01:26.479349 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:26.479282    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="f719bfad-7662-4706-aed4-c080a4912db2" containerName="payload-processing"
Jul 29 05:01:26.479349 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:26.479288    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="8dbc9aec-d52f-425d-b506-04eefcd2dcc0" containerName="cleanup"
Jul 29 05:01:26.479349 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:26.479293    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="182fdd57-ce2f-4ca0-acc3-a781eb32b6e5" containerName="maas-api"
Jul 29 05:01:26.479349 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:26.479298    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="69c5c2d6-a4dc-4108-91f5-e1fb2dd7234b" containerName="maas-api"
Jul 29 05:01:26.479349 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:26.479303    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="1236f357-0732-4ddf-b029-e6fb242df648" containerName="cleanup"
Jul 29 05:01:26.483292 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:26.483277    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b550f6-8kbl2"
Jul 29 05:01:26.485835 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:26.485802    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 29 05:01:26.486012 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:26.485802    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8frm\""
Jul 29 05:01:26.489668 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:26.489638    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b550f6-8kbl2"]
Jul 29 05:01:26.556699 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:26.556662    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b5c4b2d-975b-4124-9cb3-1b19a29aa338-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-b550f6-8kbl2\" (UID: \"7b5c4b2d-975b-4124-9cb3-1b19a29aa338\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b550f6-8kbl2"
Jul 29 05:01:26.658301 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:26.658270    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b5c4b2d-975b-4124-9cb3-1b19a29aa338-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-b550f6-8kbl2\" (UID: \"7b5c4b2d-975b-4124-9cb3-1b19a29aa338\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b550f6-8kbl2"
Jul 29 05:01:26.658992 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:26.658970    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b5c4b2d-975b-4124-9cb3-1b19a29aa338-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-b550f6-8kbl2\" (UID: \"7b5c4b2d-975b-4124-9cb3-1b19a29aa338\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b550f6-8kbl2"
Jul 29 05:01:26.795334 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:26.795244    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b550f6-8kbl2"
Jul 29 05:01:26.931045 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:26.931022    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b550f6-8kbl2"]
Jul 29 05:01:26.933447 ip-10-0-135-138 kubenswrapper[2570]: W0729 05:01:26.933422    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b5c4b2d_975b_4124_9cb3_1b19a29aa338.slice/crio-db1dabd20bbeb30a9874f682675182ef0a92edc3a6d05b98d0b06c7bf8c7277b WatchSource:0}: Error finding container db1dabd20bbeb30a9874f682675182ef0a92edc3a6d05b98d0b06c7bf8c7277b: Status 404 returned error can't find the container with id db1dabd20bbeb30a9874f682675182ef0a92edc3a6d05b98d0b06c7bf8c7277b
Jul 29 05:01:27.637038 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:27.637003    2570 generic.go:356] "Generic (PLEG): container finished" podID="7b5c4b2d-975b-4124-9cb3-1b19a29aa338" containerID="bfa9ff357cc0e1c2409ee16a9f87878df414ad7f0643e0ba1032652c685ed2f5" exitCode=0
Jul 29 05:01:27.637352 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:27.637084    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b550f6-8kbl2" event={"ID":"7b5c4b2d-975b-4124-9cb3-1b19a29aa338","Type":"ContainerDied","Data":"bfa9ff357cc0e1c2409ee16a9f87878df414ad7f0643e0ba1032652c685ed2f5"}
Jul 29 05:01:27.637352 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:27.637155    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b550f6-8kbl2" event={"ID":"7b5c4b2d-975b-4124-9cb3-1b19a29aa338","Type":"ContainerStarted","Data":"db1dabd20bbeb30a9874f682675182ef0a92edc3a6d05b98d0b06c7bf8c7277b"}
Jul 29 05:01:28.775592 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:28.775565    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b550f6-8kbl2"
Jul 29 05:01:28.875472 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:28.875441    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b5c4b2d-975b-4124-9cb3-1b19a29aa338-maas-api-service-ca\") pod \"7b5c4b2d-975b-4124-9cb3-1b19a29aa338\" (UID: \"7b5c4b2d-975b-4124-9cb3-1b19a29aa338\") "
Jul 29 05:01:28.875841 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:28.875795    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b5c4b2d-975b-4124-9cb3-1b19a29aa338-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7b5c4b2d-975b-4124-9cb3-1b19a29aa338" (UID: "7b5c4b2d-975b-4124-9cb3-1b19a29aa338"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 05:01:28.976912 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:28.976809    2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b5c4b2d-975b-4124-9cb3-1b19a29aa338-maas-api-service-ca\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 05:01:29.648146 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:29.648118    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b550f6-8kbl2"
Jul 29 05:01:29.648326 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:29.648161    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b550f6-8kbl2" event={"ID":"7b5c4b2d-975b-4124-9cb3-1b19a29aa338","Type":"ContainerDied","Data":"db1dabd20bbeb30a9874f682675182ef0a92edc3a6d05b98d0b06c7bf8c7277b"}
Jul 29 05:01:29.648326 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:29.648204    2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db1dabd20bbeb30a9874f682675182ef0a92edc3a6d05b98d0b06c7bf8c7277b"
Jul 29 05:01:36.889351 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:36.889311    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-b550f6-29755020-kb668"]
Jul 29 05:01:36.891187 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:36.891161    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-b550f6-29755020-kb668"]
Jul 29 05:01:36.932909 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:36.932879    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-b550f6-6c8767bd9b-z4lkb"]
Jul 29 05:01:36.933154 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:36.933128    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-b550f6-6c8767bd9b-z4lkb" podUID="3e7e436c-94f3-47fb-b155-9ec558a4ecd1" containerName="payload-pre-processing" containerID="cri-o://328dd955241340e54759d170034900538e5947bef565e0e46f2e544a97ec3c8f" gracePeriod=30
Jul 29 05:01:37.091881 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:37.091838    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b550f6-8kbl2"]
Jul 29 05:01:37.094219 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:37.094191    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b550f6-8kbl2"]
Jul 29 05:01:37.222248 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:37.222221    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-b550f6-6c8767bd9b-z4lkb"
Jul 29 05:01:37.359734 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:37.359695    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n2bc9\" (UniqueName: \"kubernetes.io/projected/3e7e436c-94f3-47fb-b155-9ec558a4ecd1-kube-api-access-n2bc9\") pod \"3e7e436c-94f3-47fb-b155-9ec558a4ecd1\" (UID: \"3e7e436c-94f3-47fb-b155-9ec558a4ecd1\") "
Jul 29 05:01:37.359963 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:37.359855    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e7e436c-94f3-47fb-b155-9ec558a4ecd1-plugins-config-volume\") pod \"3e7e436c-94f3-47fb-b155-9ec558a4ecd1\" (UID: \"3e7e436c-94f3-47fb-b155-9ec558a4ecd1\") "
Jul 29 05:01:37.360463 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:37.360434    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e7e436c-94f3-47fb-b155-9ec558a4ecd1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3e7e436c-94f3-47fb-b155-9ec558a4ecd1" (UID: "3e7e436c-94f3-47fb-b155-9ec558a4ecd1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 05:01:37.362713 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:37.362682    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e7e436c-94f3-47fb-b155-9ec558a4ecd1-kube-api-access-n2bc9" (OuterVolumeSpecName: "kube-api-access-n2bc9") pod "3e7e436c-94f3-47fb-b155-9ec558a4ecd1" (UID: "3e7e436c-94f3-47fb-b155-9ec558a4ecd1"). InnerVolumeSpecName "kube-api-access-n2bc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 05:01:37.461575 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:37.461504    2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e7e436c-94f3-47fb-b155-9ec558a4ecd1-plugins-config-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 05:01:37.461575 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:37.461531    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2bc9\" (UniqueName: \"kubernetes.io/projected/3e7e436c-94f3-47fb-b155-9ec558a4ecd1-kube-api-access-n2bc9\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Jul 29 05:01:37.688777 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:37.688741    2570 generic.go:356] "Generic (PLEG): container finished" podID="3e7e436c-94f3-47fb-b155-9ec558a4ecd1" containerID="328dd955241340e54759d170034900538e5947bef565e0e46f2e544a97ec3c8f" exitCode=0
Jul 29 05:01:37.688979 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:37.688784    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-b550f6-6c8767bd9b-z4lkb"
Jul 29 05:01:37.688979 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:37.688841    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-b550f6-6c8767bd9b-z4lkb" event={"ID":"3e7e436c-94f3-47fb-b155-9ec558a4ecd1","Type":"ContainerDied","Data":"328dd955241340e54759d170034900538e5947bef565e0e46f2e544a97ec3c8f"}
Jul 29 05:01:37.688979 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:37.688894    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-b550f6-6c8767bd9b-z4lkb" event={"ID":"3e7e436c-94f3-47fb-b155-9ec558a4ecd1","Type":"ContainerDied","Data":"3fcc64939cd9c886b10735ca6f263fd029bfe2825d41195a8159551d5dc2e4be"}
Jul 29 05:01:37.688979 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:37.688917    2570 scope.go:117] "RemoveContainer" containerID="328dd955241340e54759d170034900538e5947bef565e0e46f2e544a97ec3c8f"
Jul 29 05:01:37.697690 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:37.697669    2570 scope.go:117] "RemoveContainer" containerID="328dd955241340e54759d170034900538e5947bef565e0e46f2e544a97ec3c8f"
Jul 29 05:01:37.697959 ip-10-0-135-138 kubenswrapper[2570]: E0729 05:01:37.697942    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"328dd955241340e54759d170034900538e5947bef565e0e46f2e544a97ec3c8f\": container with ID starting with 328dd955241340e54759d170034900538e5947bef565e0e46f2e544a97ec3c8f not found: ID does not exist" containerID="328dd955241340e54759d170034900538e5947bef565e0e46f2e544a97ec3c8f"
Jul 29 05:01:37.698012 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:37.697967    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"328dd955241340e54759d170034900538e5947bef565e0e46f2e544a97ec3c8f"} err="failed to get container status \"328dd955241340e54759d170034900538e5947bef565e0e46f2e544a97ec3c8f\": rpc error: code = NotFound desc = could not find container \"328dd955241340e54759d170034900538e5947bef565e0e46f2e544a97ec3c8f\": container with ID starting with 328dd955241340e54759d170034900538e5947bef565e0e46f2e544a97ec3c8f not found: ID does not exist"
Jul 29 05:01:37.716634 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:37.716565    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-b550f6-6c8767bd9b-z4lkb"]
Jul 29 05:01:37.718440 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:37.718419    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-b550f6-6c8767bd9b-z4lkb"]
Jul 29 05:01:37.836351 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:37.836319    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e7e436c-94f3-47fb-b155-9ec558a4ecd1" path="/var/lib/kubelet/pods/3e7e436c-94f3-47fb-b155-9ec558a4ecd1/volumes"
Jul 29 05:01:37.836688 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:37.836673    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b5c4b2d-975b-4124-9cb3-1b19a29aa338" path="/var/lib/kubelet/pods/7b5c4b2d-975b-4124-9cb3-1b19a29aa338/volumes"
Jul 29 05:01:37.836964 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:37.836952    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="862757c8-f510-426e-96be-61657c958aff" path="/var/lib/kubelet/pods/862757c8-f510-426e-96be-61657c958aff/volumes"
Jul 29 05:01:50.177148 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:50.177117    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d56969f4b-mdm9z_8e7db7d0-36b3-460e-8127-910806de4c83/manager/0.log"
Jul 29 05:01:50.182863 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:50.182843    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d56969f4b-mdm9z_8e7db7d0-36b3-460e-8127-910806de4c83/copy-manifests/0.log"
Jul 29 05:01:50.937174 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:50.937141    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29755020-dxr62_8dbc9aec-d52f-425d-b506-04eefcd2dcc0/cleanup/0.log"
Jul 29 05:01:52.887017 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:52.886984    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-wqt5k_0879e712-d04d-496c-8fcd-6ef1fcce6bc1/manager/0.log"
Jul 29 05:01:53.796534 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:53.796505    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-664ccd966c-brch5_87caebc4-95a0-4bd4-bd96-884de1f86d3b/kube-auth-proxy/0.log"
Jul 29 05:01:54.028571 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:54.028542    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-4qq2r_c237404f-9025-4bd0-9b69-b060923d2ef0/payload-pre-processing/0.log"
Jul 29 05:01:54.141987 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:54.141946    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-gznsm_a1ffbe6e-a508-4235-a696-01cd077b7423/payload-processing/0.log"
Jul 29 05:01:54.809738 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:54.809710    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-lxxjr_44240c25-e95a-44d8-81c0-2b87772614c4/main/0.log"
Jul 29 05:01:54.915086 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:54.915057    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs87wf_cc0069b5-b84c-48de-80ab-86cf9718cc0b/main/0.log"
Jul 29 05:01:55.134543 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:01:55.134511    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-zdg6k_7f64374e-db22-4963-bdf5-0ffce977a69e/main/0.log"
Jul 29 05:02:03.456147 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:03.456115    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-vvzjp_d9e67808-ec21-42a2-bbaf-a28164326ccc/global-pull-secret-syncer/0.log"
Jul 29 05:02:03.587353 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:03.587325    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-k5hmf_7252c41b-1992-4e77-b76d-8f036a55e064/konnectivity-agent/0.log"
Jul 29 05:02:03.664363 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:03.664337    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-138.ec2.internal_f49f38cf04909602874266f431d33228/haproxy/0.log"
Jul 29 05:02:09.356722 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:09.356695    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-wqt5k_0879e712-d04d-496c-8fcd-6ef1fcce6bc1/manager/0.log"
Jul 29 05:02:11.113846 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:11.113801    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1416a1fd-24a7-4d3c-93e7-5d8198df9cac/alertmanager/0.log"
Jul 29 05:02:11.137258 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:11.137229    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1416a1fd-24a7-4d3c-93e7-5d8198df9cac/config-reloader/0.log"
Jul 29 05:02:11.163879 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:11.163857    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1416a1fd-24a7-4d3c-93e7-5d8198df9cac/kube-rbac-proxy-web/0.log"
Jul 29 05:02:11.185020 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:11.185001    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1416a1fd-24a7-4d3c-93e7-5d8198df9cac/kube-rbac-proxy/0.log"
Jul 29 05:02:11.206194 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:11.206176    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1416a1fd-24a7-4d3c-93e7-5d8198df9cac/kube-rbac-proxy-metric/0.log"
Jul 29 05:02:11.233083 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:11.233055    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1416a1fd-24a7-4d3c-93e7-5d8198df9cac/prom-label-proxy/0.log"
Jul 29 05:02:11.254324 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:11.254250    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1416a1fd-24a7-4d3c-93e7-5d8198df9cac/init-config-reloader/0.log"
Jul 29 05:02:11.332159 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:11.332131    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-f7hkw_254912df-9a41-48da-834d-7098495e379c/kube-state-metrics/0.log"
Jul 29 05:02:11.353991 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:11.353972    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-f7hkw_254912df-9a41-48da-834d-7098495e379c/kube-rbac-proxy-main/0.log"
Jul 29 05:02:11.379027 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:11.379008    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-f7hkw_254912df-9a41-48da-834d-7098495e379c/kube-rbac-proxy-self/0.log"
Jul 29 05:02:11.405646 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:11.405625    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7f5b4b5cf8-nvzjr_aca3b4b2-ca5d-4712-9b80-d9d2a41380e6/metrics-server/0.log"
Jul 29 05:02:11.430333 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:11.430315    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-p2t7j_1caf42ad-2327-496f-8c05-a3059abcf892/monitoring-plugin/0.log"
Jul 29 05:02:11.619984 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:11.619958    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-k78tt_354094fb-89f8-4929-aae9-dc99c906d34d/node-exporter/0.log"
Jul 29 05:02:11.642801 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:11.642780    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-k78tt_354094fb-89f8-4929-aae9-dc99c906d34d/kube-rbac-proxy/0.log"
Jul 29 05:02:11.663690 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:11.663671    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-k78tt_354094fb-89f8-4929-aae9-dc99c906d34d/init-textfile/0.log"
Jul 29 05:02:11.726449 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:11.726427    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-bwtm8_364ce126-2396-443f-a420-7be5f74d1494/kube-rbac-proxy-main/0.log"
Jul 29 05:02:11.750859 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:11.750834    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-bwtm8_364ce126-2396-443f-a420-7be5f74d1494/kube-rbac-proxy-self/0.log"
Jul 29 05:02:11.779263 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:11.779240    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-bwtm8_364ce126-2396-443f-a420-7be5f74d1494/openshift-state-metrics/0.log"
Jul 29 05:02:11.817437 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:11.817420    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3cd902a6-d01f-469e-85f0-0d6fd7ff42c6/prometheus/0.log"
Jul 29 05:02:11.841992 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:11.841963    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3cd902a6-d01f-469e-85f0-0d6fd7ff42c6/config-reloader/0.log"
Jul 29 05:02:11.866032 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:11.866011    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3cd902a6-d01f-469e-85f0-0d6fd7ff42c6/thanos-sidecar/0.log"
Jul 29 05:02:11.888742 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:11.888675    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3cd902a6-d01f-469e-85f0-0d6fd7ff42c6/kube-rbac-proxy-web/0.log"
Jul 29 05:02:11.913253 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:11.913223    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3cd902a6-d01f-469e-85f0-0d6fd7ff42c6/kube-rbac-proxy/0.log"
Jul 29 05:02:11.936839 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:11.936793    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3cd902a6-d01f-469e-85f0-0d6fd7ff42c6/kube-rbac-proxy-thanos/0.log"
Jul 29 05:02:11.958507 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:11.958490    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3cd902a6-d01f-469e-85f0-0d6fd7ff42c6/init-config-reloader/0.log"
Jul 29 05:02:11.984672 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:11.984650    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-b84ft_3949548e-4b2f-4982-a946-e383513165bd/prometheus-operator/0.log"
Jul 29 05:02:12.006257 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:12.006233    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-b84ft_3949548e-4b2f-4982-a946-e383513165bd/kube-rbac-proxy/0.log"
Jul 29 05:02:12.031241 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:12.031221    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-bm98m_d130e173-40bc-4422-aeae-efe66f929eaa/prometheus-operator-admission-webhook/0.log"
Jul 29 05:02:12.218762 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:12.218692    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8686848888-grwsz_4b2f4957-a89a-4c88-beff-248aca3c6c89/thanos-query/0.log"
Jul 29 05:02:12.246179 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:12.246152    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8686848888-grwsz_4b2f4957-a89a-4c88-beff-248aca3c6c89/kube-rbac-proxy-web/0.log"
Jul 29 05:02:12.268725 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:12.268705    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8686848888-grwsz_4b2f4957-a89a-4c88-beff-248aca3c6c89/kube-rbac-proxy/0.log"
Jul 29 05:02:12.301945 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:12.301921    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8686848888-grwsz_4b2f4957-a89a-4c88-beff-248aca3c6c89/prom-label-proxy/0.log"
Jul 29 05:02:12.323101 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:12.323070    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8686848888-grwsz_4b2f4957-a89a-4c88-beff-248aca3c6c89/kube-rbac-proxy-rules/0.log"
Jul 29 05:02:12.344362 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:12.344341    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8686848888-grwsz_4b2f4957-a89a-4c88-beff-248aca3c6c89/kube-rbac-proxy-metrics/0.log"
Jul 29 05:02:13.365932 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:13.365898    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-q2sff/perf-node-gather-daemonset-778wh"]
Jul 29 05:02:13.366315 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:13.366303    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e7e436c-94f3-47fb-b155-9ec558a4ecd1" containerName="payload-pre-processing"
Jul 29 05:02:13.366315 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:13.366314    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e7e436c-94f3-47fb-b155-9ec558a4ecd1" containerName="payload-pre-processing"
Jul 29 05:02:13.366413 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:13.366326    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b5c4b2d-975b-4124-9cb3-1b19a29aa338" containerName="revoke-keys"
Jul 29 05:02:13.366413 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:13.366332    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b5c4b2d-975b-4124-9cb3-1b19a29aa338" containerName="revoke-keys"
Jul 29 05:02:13.366413 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:13.366407    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e7e436c-94f3-47fb-b155-9ec558a4ecd1" containerName="payload-pre-processing"
Jul 29 05:02:13.366413 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:13.366415    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b5c4b2d-975b-4124-9cb3-1b19a29aa338" containerName="revoke-keys"
Jul 29 05:02:13.369810 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:13.369792    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q2sff/perf-node-gather-daemonset-778wh"
Jul 29 05:02:13.372405 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:13.372383    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-q2sff\"/\"default-dockercfg-xt2ns\""
Jul 29 05:02:13.373754 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:13.373731    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-q2sff\"/\"kube-root-ca.crt\""
Jul 29 05:02:13.373892 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:13.373786    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-q2sff\"/\"openshift-service-ca.crt\""
Jul 29 05:02:13.379119 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:13.379098    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q2sff/perf-node-gather-daemonset-778wh"]
Jul 29 05:02:13.489749 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:13.489715    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrswk\" (UniqueName: \"kubernetes.io/projected/e1348598-aeb0-4784-bd8e-af91d5690bb5-kube-api-access-xrswk\") pod \"perf-node-gather-daemonset-778wh\" (UID: \"e1348598-aeb0-4784-bd8e-af91d5690bb5\") " pod="openshift-must-gather-q2sff/perf-node-gather-daemonset-778wh"
Jul 29 05:02:13.489978 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:13.489797    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e1348598-aeb0-4784-bd8e-af91d5690bb5-lib-modules\") pod \"perf-node-gather-daemonset-778wh\" (UID: \"e1348598-aeb0-4784-bd8e-af91d5690bb5\") " pod="openshift-must-gather-q2sff/perf-node-gather-daemonset-778wh"
Jul 29 05:02:13.489978 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:13.489868    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e1348598-aeb0-4784-bd8e-af91d5690bb5-proc\") pod \"perf-node-gather-daemonset-778wh\" (UID: \"e1348598-aeb0-4784-bd8e-af91d5690bb5\") " pod="openshift-must-gather-q2sff/perf-node-gather-daemonset-778wh"
Jul 29 05:02:13.489978 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:13.489900    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e1348598-aeb0-4784-bd8e-af91d5690bb5-sys\") pod \"perf-node-gather-daemonset-778wh\" (UID: \"e1348598-aeb0-4784-bd8e-af91d5690bb5\") " pod="openshift-must-gather-q2sff/perf-node-gather-daemonset-778wh"
Jul 29 05:02:13.489978 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:13.489938    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e1348598-aeb0-4784-bd8e-af91d5690bb5-podres\") pod \"perf-node-gather-daemonset-778wh\" (UID: \"e1348598-aeb0-4784-bd8e-af91d5690bb5\") " pod="openshift-must-gather-q2sff/perf-node-gather-daemonset-778wh"
Jul 29 05:02:13.590653 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:13.590625    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e1348598-aeb0-4784-bd8e-af91d5690bb5-lib-modules\") pod \"perf-node-gather-daemonset-778wh\" (UID: \"e1348598-aeb0-4784-bd8e-af91d5690bb5\") " pod="openshift-must-gather-q2sff/perf-node-gather-daemonset-778wh"
Jul 29 05:02:13.590812 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:13.590664    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e1348598-aeb0-4784-bd8e-af91d5690bb5-proc\") pod \"perf-node-gather-daemonset-778wh\" (UID: \"e1348598-aeb0-4784-bd8e-af91d5690bb5\") " pod="openshift-must-gather-q2sff/perf-node-gather-daemonset-778wh"
Jul 29 05:02:13.590812 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:13.590684    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e1348598-aeb0-4784-bd8e-af91d5690bb5-sys\") pod \"perf-node-gather-daemonset-778wh\" (UID: \"e1348598-aeb0-4784-bd8e-af91d5690bb5\") " pod="openshift-must-gather-q2sff/perf-node-gather-daemonset-778wh"
Jul 29 05:02:13.590812 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:13.590709    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e1348598-aeb0-4784-bd8e-af91d5690bb5-podres\") pod \"perf-node-gather-daemonset-778wh\" (UID: \"e1348598-aeb0-4784-bd8e-af91d5690bb5\") " pod="openshift-must-gather-q2sff/perf-node-gather-daemonset-778wh"
Jul 29 05:02:13.590812 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:13.590755    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrswk\" (UniqueName: \"kubernetes.io/projected/e1348598-aeb0-4784-bd8e-af91d5690bb5-kube-api-access-xrswk\") pod \"perf-node-gather-daemonset-778wh\" (UID: \"e1348598-aeb0-4784-bd8e-af91d5690bb5\") " pod="openshift-must-gather-q2sff/perf-node-gather-daemonset-778wh"
Jul 29 05:02:13.590812 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:13.590766    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e1348598-aeb0-4784-bd8e-af91d5690bb5-proc\") pod \"perf-node-gather-daemonset-778wh\" (UID: \"e1348598-aeb0-4784-bd8e-af91d5690bb5\") " pod="openshift-must-gather-q2sff/perf-node-gather-daemonset-778wh"
Jul 29 05:02:13.590812 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:13.590771    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e1348598-aeb0-4784-bd8e-af91d5690bb5-sys\") pod \"perf-node-gather-daemonset-778wh\" (UID: \"e1348598-aeb0-4784-bd8e-af91d5690bb5\") " pod="openshift-must-gather-q2sff/perf-node-gather-daemonset-778wh"
Jul 29 05:02:13.591070 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:13.590848    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e1348598-aeb0-4784-bd8e-af91d5690bb5-lib-modules\") pod \"perf-node-gather-daemonset-778wh\" (UID: \"e1348598-aeb0-4784-bd8e-af91d5690bb5\") " pod="openshift-must-gather-q2sff/perf-node-gather-daemonset-778wh"
Jul 29 05:02:13.591070 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:13.590864    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e1348598-aeb0-4784-bd8e-af91d5690bb5-podres\") pod \"perf-node-gather-daemonset-778wh\" (UID: \"e1348598-aeb0-4784-bd8e-af91d5690bb5\") " pod="openshift-must-gather-q2sff/perf-node-gather-daemonset-778wh"
Jul 29 05:02:13.600651 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:13.600631    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrswk\" (UniqueName: \"kubernetes.io/projected/e1348598-aeb0-4784-bd8e-af91d5690bb5-kube-api-access-xrswk\") pod \"perf-node-gather-daemonset-778wh\" (UID: \"e1348598-aeb0-4784-bd8e-af91d5690bb5\") " pod="openshift-must-gather-q2sff/perf-node-gather-daemonset-778wh"
Jul 29 05:02:13.680070 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:13.680008    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q2sff/perf-node-gather-daemonset-778wh"
Jul 29 05:02:13.809146 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:13.809115    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q2sff/perf-node-gather-daemonset-778wh"]
Jul 29 05:02:13.811077 ip-10-0-135-138 kubenswrapper[2570]: W0729 05:02:13.811036    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode1348598_aeb0_4784_bd8e_af91d5690bb5.slice/crio-a3369c2ccfed737b26b18683243d975ac25f6c3f52fb5b3cc1ca218693279a38 WatchSource:0}: Error finding container a3369c2ccfed737b26b18683243d975ac25f6c3f52fb5b3cc1ca218693279a38: Status 404 returned error can't find the container with id a3369c2ccfed737b26b18683243d975ac25f6c3f52fb5b3cc1ca218693279a38
Jul 29 05:02:13.873278 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:13.873254    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q2sff/perf-node-gather-daemonset-778wh" event={"ID":"e1348598-aeb0-4784-bd8e-af91d5690bb5","Type":"ContainerStarted","Data":"dba39fb9d43fde8b12967a51944fe70186dbccae7a143c6eeed21c60724fe0d4"}
Jul 29 05:02:13.873390 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:13.873302    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q2sff/perf-node-gather-daemonset-778wh" event={"ID":"e1348598-aeb0-4784-bd8e-af91d5690bb5","Type":"ContainerStarted","Data":"a3369c2ccfed737b26b18683243d975ac25f6c3f52fb5b3cc1ca218693279a38"}
Jul 29 05:02:13.873390 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:13.873328    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-q2sff/perf-node-gather-daemonset-778wh"
Jul 29 05:02:13.894012 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:13.890137    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-q2sff/perf-node-gather-daemonset-778wh" podStartSLOduration=0.890118288 podStartE2EDuration="890.118288ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 05:02:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 05:02:13.887714116 +0000 UTC m=+3998.617845188" watchObservedRunningTime="2026-07-29 05:02:13.890118288 +0000 UTC m=+3998.620249361"
Jul 29 05:02:14.586008 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:14.585973    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-8677fddb6d-qrwsn_c5695ea5-b8ee-4dbd-a7d4-e1a97ef0cf8b/console/0.log"
Jul 29 05:02:16.164347 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:16.164308    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9rmhw_30b69d69-48d9-48a7-ad3c-dad8542537ee/dns/0.log"
Jul 29 05:02:16.184844 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:16.184801    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9rmhw_30b69d69-48d9-48a7-ad3c-dad8542537ee/kube-rbac-proxy/0.log"
Jul 29 05:02:16.315937 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:16.315907    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hc29h_c00b9ff9-d986-4043-8403-c8d9a68596da/dns-node-resolver/0.log"
Jul 29 05:02:16.916452 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:16.916426    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kkpf7_7b3ec617-26f1-4c28-aa08-2710238467ac/node-ca/0.log"
Jul 29 05:02:18.038193 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:18.038165    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-664ccd966c-brch5_87caebc4-95a0-4bd4-bd96-884de1f86d3b/kube-auth-proxy/0.log"
Jul 29 05:02:18.142538 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:18.142515    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-4qq2r_c237404f-9025-4bd0-9b69-b060923d2ef0/payload-pre-processing/0.log"
Jul 29 05:02:18.173559 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:18.173534    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-gznsm_a1ffbe6e-a508-4235-a696-01cd077b7423/payload-processing/0.log"
Jul 29 05:02:18.995245 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:18.995213    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-nhdcg_00b8825f-8f3b-4da7-929c-df23b76f32d8/serve-healthcheck-canary/0.log"
Jul 29 05:02:19.649800 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:19.649769    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mf7ck_55fa3eb7-80e7-4b52-bdbe-766ae9a5b93f/kube-rbac-proxy/0.log"
Jul 29 05:02:19.671048 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:19.671028    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mf7ck_55fa3eb7-80e7-4b52-bdbe-766ae9a5b93f/exporter/0.log"
Jul 29 05:02:19.692048 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:19.692024    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mf7ck_55fa3eb7-80e7-4b52-bdbe-766ae9a5b93f/extractor/0.log"
Jul 29 05:02:19.888495 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:19.888471    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-q2sff/perf-node-gather-daemonset-778wh"
Jul 29 05:02:22.559080 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:22.559051    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d56969f4b-mdm9z_8e7db7d0-36b3-460e-8127-910806de4c83/manager/0.log"
Jul 29 05:02:22.582338 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:22.582315    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d56969f4b-mdm9z_8e7db7d0-36b3-460e-8127-910806de4c83/copy-manifests/0.log"
Jul 29 05:02:24.015931 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:24.015902    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-86cc57cc46-qjlpt_a9290a1c-bb9c-4b67-b357-8ff1be3fbeda/manager/0.log"
Jul 29 05:02:24.668726 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:24.668645    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-664ccd966c-brch5_87caebc4-95a0-4bd4-bd96-884de1f86d3b/kube-auth-proxy/0.log"
Jul 29 05:02:24.751559 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:24.751521    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-4qq2r_c237404f-9025-4bd0-9b69-b060923d2ef0/payload-pre-processing/0.log"
Jul 29 05:02:24.770623 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:24.770592    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-gznsm_a1ffbe6e-a508-4235-a696-01cd077b7423/payload-processing/0.log"
Jul 29 05:02:26.290951 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:26.290925    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-664ccd966c-brch5_87caebc4-95a0-4bd4-bd96-884de1f86d3b/kube-auth-proxy/0.log"
Jul 29 05:02:26.378064 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:26.378031    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-4qq2r_c237404f-9025-4bd0-9b69-b060923d2ef0/payload-pre-processing/0.log"
Jul 29 05:02:26.397693 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:26.397666    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-gznsm_a1ffbe6e-a508-4235-a696-01cd077b7423/payload-processing/0.log"
Jul 29 05:02:30.475008 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:30.474981    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5ddp8_a5135b47-d2a7-45d5-ab3b-cb4404d1c00b/kube-multus-additional-cni-plugins/0.log"
Jul 29 05:02:30.496598 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:30.496574    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5ddp8_a5135b47-d2a7-45d5-ab3b-cb4404d1c00b/egress-router-binary-copy/0.log"
Jul 29 05:02:30.518146 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:30.518121    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5ddp8_a5135b47-d2a7-45d5-ab3b-cb4404d1c00b/cni-plugins/0.log"
Jul 29 05:02:30.538558 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:30.538538    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5ddp8_a5135b47-d2a7-45d5-ab3b-cb4404d1c00b/bond-cni-plugin/0.log"
Jul 29 05:02:30.558276 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:30.558244    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5ddp8_a5135b47-d2a7-45d5-ab3b-cb4404d1c00b/routeoverride-cni/0.log"
Jul 29 05:02:30.578365 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:30.578350    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5ddp8_a5135b47-d2a7-45d5-ab3b-cb4404d1c00b/whereabouts-cni-bincopy/0.log"
Jul 29 05:02:30.598329 ip-10-0-135-138 kubenswrapper[2570]: I0729 05:02:30.598304    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5ddp8_a5135b47-d2a7-45d5-ab3b-cb4404d1c00b/whereabouts-cni/0.log"