Jul 26 15:50:07.786022 ip-10-0-133-204 systemd[1]: Starting Kubernetes Kubelet...
Jul 26 15:50:08.162023 ip-10-0-133-204 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 26 15:50:08.162023 ip-10-0-133-204 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 26 15:50:08.162023 ip-10-0-133-204 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 26 15:50:08.162023 ip-10-0-133-204 kubenswrapper[2563]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 26 15:50:08.162023 ip-10-0-133-204 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 26 15:50:08.162941 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.162851    2563 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 26 15:50:08.167218 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167199    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 26 15:50:08.167218 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167217    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 26 15:50:08.167218 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167222    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 26 15:50:08.167218 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167226    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 26 15:50:08.167444 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167233    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 26 15:50:08.167444 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167238    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 26 15:50:08.167444 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167242    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 26 15:50:08.167444 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167247    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 26 15:50:08.167444 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167251    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 26 15:50:08.167444 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167255    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 26 15:50:08.167444 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167260    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 26 15:50:08.167444 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167264    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 26 15:50:08.167444 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167268    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 26 15:50:08.167444 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167271    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 26 15:50:08.167444 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167275    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 26 15:50:08.167444 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167281    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 26 15:50:08.167444 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167286    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 26 15:50:08.167444 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167291    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 26 15:50:08.167444 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167295    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 26 15:50:08.167444 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167306    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 26 15:50:08.167444 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167310    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 26 15:50:08.167444 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167315    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 26 15:50:08.167444 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167319    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 26 15:50:08.168050 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167323    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 26 15:50:08.168050 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167327    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 26 15:50:08.168050 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167331    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 26 15:50:08.168050 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167336    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 26 15:50:08.168050 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167340    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 26 15:50:08.168050 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167345    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 26 15:50:08.168050 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167349    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 26 15:50:08.168050 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167354    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 26 15:50:08.168050 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167358    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 26 15:50:08.168050 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167364    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 26 15:50:08.168050 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167368    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 26 15:50:08.168050 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167372    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 26 15:50:08.168050 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167376    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 26 15:50:08.168050 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167380    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 26 15:50:08.168050 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167385    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 26 15:50:08.168050 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167389    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 26 15:50:08.168050 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167393    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 26 15:50:08.168050 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167398    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 26 15:50:08.168050 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167402    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 26 15:50:08.168050 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167406    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 26 15:50:08.168519 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167410    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 26 15:50:08.168519 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167414    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 26 15:50:08.168519 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167418    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 26 15:50:08.168519 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167423    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 26 15:50:08.168519 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167427    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 26 15:50:08.168519 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167431    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 26 15:50:08.168519 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167435    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 26 15:50:08.168519 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167440    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 26 15:50:08.168519 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167445    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 26 15:50:08.168519 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167449    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 26 15:50:08.168519 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167455    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 26 15:50:08.168519 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167459    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 26 15:50:08.168519 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167463    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 26 15:50:08.168519 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167468    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 26 15:50:08.168519 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167472    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 26 15:50:08.168519 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167477    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 26 15:50:08.168519 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167481    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 26 15:50:08.168519 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167485    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 26 15:50:08.168519 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167489    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 26 15:50:08.168519 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167494    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 26 15:50:08.169010 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167498    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 26 15:50:08.169010 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167502    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 26 15:50:08.169010 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167506    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 26 15:50:08.169010 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167510    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 26 15:50:08.169010 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167514    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 26 15:50:08.169010 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167518    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 26 15:50:08.169010 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167523    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 26 15:50:08.169010 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167527    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 26 15:50:08.169010 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167531    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 26 15:50:08.169010 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167535    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 26 15:50:08.169010 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167539    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 26 15:50:08.169010 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167543    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 26 15:50:08.169010 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167549    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 26 15:50:08.169010 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167553    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 26 15:50:08.169010 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167558    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 26 15:50:08.169010 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167562    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 26 15:50:08.169010 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167566    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 26 15:50:08.169010 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167570    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 26 15:50:08.169010 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167574    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 26 15:50:08.169010 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167578    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 26 15:50:08.169511 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167582    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 26 15:50:08.169511 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167586    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 26 15:50:08.169511 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167590    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 26 15:50:08.169511 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167595    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 26 15:50:08.169511 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167599    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 26 15:50:08.169511 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167603    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 26 15:50:08.169511 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167607    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 26 15:50:08.169511 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167612    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 26 15:50:08.169511 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167616    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 26 15:50:08.169511 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167621    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 26 15:50:08.169511 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167625    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 26 15:50:08.169511 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167629    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 26 15:50:08.169511 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167633    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 26 15:50:08.169511 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167638    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 26 15:50:08.169511 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167642    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 26 15:50:08.169511 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.167647    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 26 15:50:08.169511 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169041    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 26 15:50:08.169511 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169050    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 26 15:50:08.169511 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169053    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 26 15:50:08.169971 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169056    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 26 15:50:08.169971 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169059    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 26 15:50:08.169971 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169062    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 26 15:50:08.169971 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169065    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 26 15:50:08.169971 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169068    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 26 15:50:08.169971 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169070    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 26 15:50:08.169971 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169074    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 26 15:50:08.169971 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169077    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 26 15:50:08.169971 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169080    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 26 15:50:08.169971 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169082    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 26 15:50:08.169971 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169085    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 26 15:50:08.169971 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169087    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 26 15:50:08.169971 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169089    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 26 15:50:08.169971 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169092    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 26 15:50:08.169971 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169094    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 26 15:50:08.169971 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169097    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 26 15:50:08.169971 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169100    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 26 15:50:08.169971 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169102    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 26 15:50:08.169971 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169105    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 26 15:50:08.170429 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169107    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 26 15:50:08.170429 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169109    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 26 15:50:08.170429 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169112    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 26 15:50:08.170429 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169114    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 26 15:50:08.170429 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169117    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 26 15:50:08.170429 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169119    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 26 15:50:08.170429 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169122    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 26 15:50:08.170429 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169125    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 26 15:50:08.170429 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169127    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 26 15:50:08.170429 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169129    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 26 15:50:08.170429 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169132    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 26 15:50:08.170429 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169135    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 26 15:50:08.170429 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169138    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 26 15:50:08.170429 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169140    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 26 15:50:08.170429 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169143    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 26 15:50:08.170429 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169145    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 26 15:50:08.170429 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169147    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 26 15:50:08.170429 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169150    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 26 15:50:08.170429 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169152    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 26 15:50:08.170429 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169155    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 26 15:50:08.170927 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169157    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 26 15:50:08.170927 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169161    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 26 15:50:08.170927 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169165    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 26 15:50:08.170927 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169168    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 26 15:50:08.170927 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169171    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 26 15:50:08.170927 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169173    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 26 15:50:08.170927 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169176    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 26 15:50:08.170927 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169178    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 26 15:50:08.170927 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169187    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 26 15:50:08.170927 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169191    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 26 15:50:08.170927 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169193    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 26 15:50:08.170927 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169197    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 26 15:50:08.170927 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169200    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 26 15:50:08.170927 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169204    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 26 15:50:08.170927 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169207    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 26 15:50:08.170927 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169210    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 26 15:50:08.170927 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169213    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 26 15:50:08.170927 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169216    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 26 15:50:08.170927 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169219    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 26 15:50:08.171369 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169223    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 26 15:50:08.171369 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169226    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 26 15:50:08.171369 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169229    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 26 15:50:08.171369 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169231    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 26 15:50:08.171369 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169235    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 26 15:50:08.171369 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169238    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 26 15:50:08.171369 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169240    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 26 15:50:08.171369 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169243    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 26 15:50:08.171369 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169246    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 26 15:50:08.171369 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169248    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 26 15:50:08.171369 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169251    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 26 15:50:08.171369 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169253    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 26 15:50:08.171369 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169256    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 26 15:50:08.171369 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169258    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 26 15:50:08.171369 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169261    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 26 15:50:08.171369 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169263    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 26 15:50:08.171369 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169266    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 26 15:50:08.171369 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169270    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 26 15:50:08.171369 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169273    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 26 15:50:08.171369 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169277    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 26 15:50:08.171857 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169280    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 26 15:50:08.171857 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169285    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 26 15:50:08.171857 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169289    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 26 15:50:08.171857 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169292    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 26 15:50:08.171857 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169295    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 26 15:50:08.171857 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169298    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 26 15:50:08.171857 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169300    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 26 15:50:08.171857 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169303    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 26 15:50:08.171857 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169306    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 26 15:50:08.171857 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169308    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 26 15:50:08.171857 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169311    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 26 15:50:08.171857 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169314    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 26 15:50:08.171857 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169316    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 26 15:50:08.171857 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169319    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 26 15:50:08.171857 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169322    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 26 15:50:08.171857 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169324    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 26 15:50:08.171857 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169326    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 26 15:50:08.171857 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169330    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 26 15:50:08.171857 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169694    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 26 15:50:08.171857 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169699    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 26 15:50:08.172334 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169703    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 26 15:50:08.172334 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169707    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 26 15:50:08.172334 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169710    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 26 15:50:08.172334 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169713    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 26 15:50:08.172334 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169716    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 26 15:50:08.172334 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169718    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 26 15:50:08.172334 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169726    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 26 15:50:08.172334 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169729    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 26 15:50:08.172334 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169732    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 26 15:50:08.172334 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169735    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 26 15:50:08.172334 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169737    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 26 15:50:08.172334 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169739    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 26 15:50:08.172334 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169742    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 26 15:50:08.172334 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169745    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 26 15:50:08.172334 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169747    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 26 15:50:08.172334 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169750    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 26 15:50:08.172334 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169752    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 26 15:50:08.172334 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169755    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 26 15:50:08.172334 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169757    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 26 15:50:08.172800 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169773    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 26 15:50:08.172800 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169776    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 26 15:50:08.172800 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169778    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 26 15:50:08.172800 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169781    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 26 15:50:08.172800 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169783    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 26 15:50:08.172800 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169786    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 26 15:50:08.172800 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169788    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 26 15:50:08.172800 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169791    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 26 15:50:08.172800 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169794    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 26 15:50:08.172800 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169798    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 26 15:50:08.172800 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169801    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 26 15:50:08.172800 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169804    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 26 15:50:08.172800 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169807    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 26 15:50:08.172800 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169809    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 26 15:50:08.172800 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169812    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 26 15:50:08.172800 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169814    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 26 15:50:08.172800 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169816    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 26 15:50:08.172800 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169819    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 26 15:50:08.172800 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169821    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 26 15:50:08.172800 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169824    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 26 15:50:08.173327 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169831    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 26 15:50:08.173327 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169834    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 26 15:50:08.173327 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169837    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 26 15:50:08.173327 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169840    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 26 15:50:08.173327 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169842    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 26 15:50:08.173327 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169844    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 26 15:50:08.173327 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169847    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 26 15:50:08.173327 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169850    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 26 15:50:08.173327 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169852    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 26 15:50:08.173327 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169855    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 26 15:50:08.173327 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169857    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 26 15:50:08.173327 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169860    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 26 15:50:08.173327 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169862    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 26 15:50:08.173327 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169865    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 26 15:50:08.173327 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169867    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 26 15:50:08.173327 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169869    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 26 15:50:08.173327 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169871    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 26 15:50:08.173327 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169874    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 26 15:50:08.173327 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169877    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 26 15:50:08.173327 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169879    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 26 15:50:08.173896 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169882    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 26 15:50:08.173896 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169884    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 26 15:50:08.173896 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169887    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 26 15:50:08.173896 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169889    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 26 15:50:08.173896 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169892    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 26 15:50:08.173896 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169894    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 26 15:50:08.173896 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169896    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 26 15:50:08.173896 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169898    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 26 15:50:08.173896 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169901    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 26 15:50:08.173896 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169903    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 26 15:50:08.173896 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169905    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 26 15:50:08.173896 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169908    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 26 15:50:08.173896 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169910    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 26 15:50:08.173896 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169918    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 26 15:50:08.173896 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169921    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 26 15:50:08.173896 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169923    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 26 15:50:08.173896 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169926    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 26 15:50:08.173896 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169928    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 26 15:50:08.173896 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169931    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 26 15:50:08.173896 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169934    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 26 15:50:08.174424 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169936    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 26 15:50:08.174424 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169939    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 26 15:50:08.174424 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169941    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 26 15:50:08.174424 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169943    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 26 15:50:08.174424 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169946    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 26 15:50:08.174424 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169948    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 26 15:50:08.174424 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169950    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 26 15:50:08.174424 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169953    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 26 15:50:08.174424 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169955    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 26 15:50:08.174424 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169958    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 26 15:50:08.174424 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169963    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 26 15:50:08.174424 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169966    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 26 15:50:08.174424 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169969    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 26 15:50:08.174424 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169972    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 26 15:50:08.174424 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169975    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 26 15:50:08.174424 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169978    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 26 15:50:08.174424 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169982    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 26 15:50:08.174424 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.169985    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 26 15:50:08.174424 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170056    2563 flags.go:64] FLAG: --address="0.0.0.0"
Jul 26 15:50:08.174424 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170062    2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 26 15:50:08.174968 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170074    2563 flags.go:64] FLAG: --anonymous-auth="true"
Jul 26 15:50:08.174968 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170086    2563 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 26 15:50:08.174968 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170090    2563 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 26 15:50:08.174968 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170093    2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 26 15:50:08.174968 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170097    2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 26 15:50:08.174968 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170101    2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 26 15:50:08.174968 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170110    2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 26 15:50:08.174968 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170112    2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 26 15:50:08.174968 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170116    2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 26 15:50:08.174968 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170119    2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 26 15:50:08.174968 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170122    2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 26 15:50:08.174968 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170124    2563 flags.go:64] FLAG: --cgroup-root=""
Jul 26 15:50:08.174968 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170127    2563 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 26 15:50:08.174968 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170130    2563 flags.go:64] FLAG: --client-ca-file=""
Jul 26 15:50:08.174968 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170133    2563 flags.go:64] FLAG: --cloud-provider="external"
Jul 26 15:50:08.174968 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170135    2563 flags.go:64] FLAG: --cluster-dns="[]"
Jul 26 15:50:08.174968 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170141    2563 flags.go:64] FLAG: --cluster-domain=""
Jul 26 15:50:08.174968 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170144    2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 26 15:50:08.174968 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170147    2563 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 26 15:50:08.174968 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170150    2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 26 15:50:08.174968 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170153    2563 flags.go:64] FLAG: --container-log-max-files="5"
Jul 26 15:50:08.174968 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170157    2563 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 26 15:50:08.174968 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170160    2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 26 15:50:08.175542 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170163    2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 26 15:50:08.175542 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170166    2563 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 26 15:50:08.175542 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170169    2563 flags.go:64] FLAG: --contention-profiling="false"
Jul 26 15:50:08.175542 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170172    2563 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 26 15:50:08.175542 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170175    2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 26 15:50:08.175542 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170179    2563 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 26 15:50:08.175542 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170182    2563 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 26 15:50:08.175542 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170186    2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 26 15:50:08.175542 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170189    2563 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 26 15:50:08.175542 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170191    2563 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 26 15:50:08.175542 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170194    2563 flags.go:64] FLAG: --enable-load-reader="false"
Jul 26 15:50:08.175542 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170197    2563 flags.go:64] FLAG: --enable-server="true"
Jul 26 15:50:08.175542 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170200    2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 26 15:50:08.175542 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170207    2563 flags.go:64] FLAG: --event-burst="100"
Jul 26 15:50:08.175542 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170209    2563 flags.go:64] FLAG: --event-qps="50"
Jul 26 15:50:08.175542 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170212    2563 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 26 15:50:08.175542 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170220    2563 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 26 15:50:08.175542 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170223    2563 flags.go:64] FLAG: --eviction-hard=""
Jul 26 15:50:08.175542 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170227    2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 26 15:50:08.175542 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170230    2563 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 26 15:50:08.175542 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170236    2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 26 15:50:08.175542 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170239    2563 flags.go:64] FLAG: --eviction-soft=""
Jul 26 15:50:08.175542 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170242    2563 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 26 15:50:08.175542 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170245    2563 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 26 15:50:08.175542 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170248    2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 26 15:50:08.176178 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170251    2563 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 26 15:50:08.176178 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170254    2563 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 26 15:50:08.176178 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170257    2563 flags.go:64] FLAG: --fail-swap-on="true"
Jul 26 15:50:08.176178 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170259    2563 flags.go:64] FLAG: --feature-gates=""
Jul 26 15:50:08.176178 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170265    2563 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 26 15:50:08.176178 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170267    2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 26 15:50:08.176178 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170270    2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 26 15:50:08.176178 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170273    2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 26 15:50:08.176178 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170276    2563 flags.go:64] FLAG: --healthz-port="10248"
Jul 26 15:50:08.176178 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170279    2563 flags.go:64] FLAG: --help="false"
Jul 26 15:50:08.176178 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170282    2563 flags.go:64] FLAG: --hostname-override="ip-10-0-133-204.ec2.internal"
Jul 26 15:50:08.176178 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170285    2563 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 26 15:50:08.176178 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170289    2563 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 26 15:50:08.176178 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170292    2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 26 15:50:08.176178 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170295    2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 26 15:50:08.176178 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170298    2563 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 26 15:50:08.176178 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170301    2563 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 26 15:50:08.176178 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170304    2563 flags.go:64] FLAG: --image-service-endpoint=""
Jul 26 15:50:08.176178 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170306    2563 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 26 15:50:08.176178 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170309    2563 flags.go:64] FLAG: --kube-api-burst="100"
Jul 26 15:50:08.176178 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170312    2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 26 15:50:08.176178 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170315    2563 flags.go:64] FLAG: --kube-api-qps="50"
Jul 26 15:50:08.176178 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170317    2563 flags.go:64] FLAG: --kube-reserved=""
Jul 26 15:50:08.176178 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170320    2563 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 26 15:50:08.176813 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170328    2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 26 15:50:08.176813 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170331    2563 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 26 15:50:08.176813 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170334    2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 26 15:50:08.176813 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170337    2563 flags.go:64] FLAG: --lock-file=""
Jul 26 15:50:08.176813 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170339    2563 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 26 15:50:08.176813 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170342    2563 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 26 15:50:08.176813 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170345    2563 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 26 15:50:08.176813 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170349    2563 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 26 15:50:08.176813 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170352    2563 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 26 15:50:08.176813 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170355    2563 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 26 15:50:08.176813 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170358    2563 flags.go:64] FLAG: --logging-format="text"
Jul 26 15:50:08.176813 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170360    2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 26 15:50:08.176813 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170365    2563 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 26 15:50:08.176813 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170368    2563 flags.go:64] FLAG: --manifest-url=""
Jul 26 15:50:08.176813 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170371    2563 flags.go:64] FLAG: --manifest-url-header=""
Jul 26 15:50:08.176813 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170375    2563 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 26 15:50:08.176813 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170378    2563 flags.go:64] FLAG: --max-open-files="1000000"
Jul 26 15:50:08.176813 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170382    2563 flags.go:64] FLAG: --max-pods="110"
Jul 26 15:50:08.176813 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170385    2563 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 26 15:50:08.176813 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170387    2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 26 15:50:08.176813 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170391    2563 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 26 15:50:08.176813 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170394    2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 26 15:50:08.176813 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170397    2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 26 15:50:08.176813 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170400    2563 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 26 15:50:08.176813 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170403    2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 26 15:50:08.177452 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170410    2563 flags.go:64] FLAG: --node-status-max-images="50"
Jul 26 15:50:08.177452 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170413    2563 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 26 15:50:08.177452 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170415    2563 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 26 15:50:08.177452 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170419    2563 flags.go:64] FLAG: --pod-cidr=""
Jul 26 15:50:08.177452 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170421    2563 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 26 15:50:08.177452 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170427    2563 flags.go:64] FLAG: --pod-manifest-path=""
Jul 26 15:50:08.177452 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170430    2563 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 26 15:50:08.177452 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170433    2563 flags.go:64] FLAG: --pods-per-core="0"
Jul 26 15:50:08.177452 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170441    2563 flags.go:64] FLAG: --port="10250"
Jul 26 15:50:08.177452 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170444    2563 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 26 15:50:08.177452 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170447    2563 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0fc4c3e6e0565155b"
Jul 26 15:50:08.177452 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170450    2563 flags.go:64] FLAG: --qos-reserved=""
Jul 26 15:50:08.177452 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170453    2563 flags.go:64] FLAG: --read-only-port="10255"
Jul 26 15:50:08.177452 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170455    2563 flags.go:64] FLAG: --register-node="true"
Jul 26 15:50:08.177452 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170458    2563 flags.go:64] FLAG: --register-with-taints=""
Jul 26 15:50:08.177452 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170466    2563 flags.go:64] FLAG: --registry-burst="10"
Jul 26 15:50:08.177452 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170469    2563 flags.go:64] FLAG: --registry-qps="5"
Jul 26 15:50:08.177452 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170472    2563 flags.go:64] FLAG: --reserved-cpus=""
Jul 26 15:50:08.177452 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170474    2563 flags.go:64] FLAG: --reserved-memory=""
Jul 26 15:50:08.177452 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170478    2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 26 15:50:08.177452 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170482    2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 26 15:50:08.177452 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170485    2563 flags.go:64] FLAG: --rotate-certificates="false"
Jul 26 15:50:08.177452 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170488    2563 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 26 15:50:08.177452 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170491    2563 flags.go:64] FLAG: --runonce="false"
Jul 26 15:50:08.177452 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170493    2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 26 15:50:08.178063 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170497    2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 26 15:50:08.178063 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170499    2563 flags.go:64] FLAG: --seccomp-default="false"
Jul 26 15:50:08.178063 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170504    2563 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 26 15:50:08.178063 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170506    2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 26 15:50:08.178063 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170509    2563 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 26 15:50:08.178063 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170512    2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 26 15:50:08.178063 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170515    2563 flags.go:64] FLAG: --storage-driver-password="root"
Jul 26 15:50:08.178063 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170518    2563 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 26 15:50:08.178063 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170521    2563 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 26 15:50:08.178063 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170523    2563 flags.go:64] FLAG: --storage-driver-user="root"
Jul 26 15:50:08.178063 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170526    2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 26 15:50:08.178063 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170529    2563 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 26 15:50:08.178063 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170532    2563 flags.go:64] FLAG: --system-cgroups=""
Jul 26 15:50:08.178063 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170534    2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 26 15:50:08.178063 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170545    2563 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 26 15:50:08.178063 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170547    2563 flags.go:64] FLAG: --tls-cert-file=""
Jul 26 15:50:08.178063 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170556    2563 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 26 15:50:08.178063 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170562    2563 flags.go:64] FLAG: --tls-min-version=""
Jul 26 15:50:08.178063 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170565    2563 flags.go:64] FLAG: --tls-private-key-file=""
Jul 26 15:50:08.178063 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170567    2563 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 26 15:50:08.178063 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170570    2563 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 26 15:50:08.178063 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170573    2563 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 26 15:50:08.178063 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170576    2563 flags.go:64] FLAG: --v="2"
Jul 26 15:50:08.178063 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170579    2563 flags.go:64] FLAG: --version="false"
Jul 26 15:50:08.178063 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170583    2563 flags.go:64] FLAG: --vmodule=""
Jul 26 15:50:08.178662 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170586    2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 26 15:50:08.178662 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.170589    2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 26 15:50:08.178662 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170718    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 26 15:50:08.178662 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170723    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 26 15:50:08.178662 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170726    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 26 15:50:08.178662 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170730    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 26 15:50:08.178662 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170732    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 26 15:50:08.178662 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170735    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 26 15:50:08.178662 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170738    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 26 15:50:08.178662 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170743    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 26 15:50:08.178662 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170748    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 26 15:50:08.178662 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170751    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 26 15:50:08.178662 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170755    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 26 15:50:08.178662 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170771    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 26 15:50:08.178662 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170774    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 26 15:50:08.178662 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170777    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 26 15:50:08.178662 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170779    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 26 15:50:08.178662 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170782    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 26 15:50:08.178662 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170784    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 26 15:50:08.178662 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170787    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 26 15:50:08.179163 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170789    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 26 15:50:08.179163 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170792    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 26 15:50:08.179163 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170794    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 26 15:50:08.179163 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170797    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 26 15:50:08.179163 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170800    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 26 15:50:08.179163 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170803    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 26 15:50:08.179163 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170805    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 26 15:50:08.179163 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170808    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 26 15:50:08.179163 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170810    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 26 15:50:08.179163 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170813    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 26 15:50:08.179163 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170815    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 26 15:50:08.179163 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170818    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 26 15:50:08.179163 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170820    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 26 15:50:08.179163 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170822    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 26 15:50:08.179163 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170826    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 26 15:50:08.179163 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170829    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 26 15:50:08.179163 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170831    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 26 15:50:08.179163 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170833    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 26 15:50:08.179163 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170836    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 26 15:50:08.179163 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170838    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 26 15:50:08.179637 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170841    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 26 15:50:08.179637 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170845    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 26 15:50:08.179637 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170848    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 26 15:50:08.179637 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170850    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 26 15:50:08.179637 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170853    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 26 15:50:08.179637 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170855    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 26 15:50:08.179637 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170858    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 26 15:50:08.179637 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170860    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 26 15:50:08.179637 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170863    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 26 15:50:08.179637 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170865    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 26 15:50:08.179637 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170868    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 26 15:50:08.179637 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170870    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 26 15:50:08.179637 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170873    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 26 15:50:08.179637 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170875    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 26 15:50:08.179637 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170878    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 26 15:50:08.179637 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170880    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 26 15:50:08.179637 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170883    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 26 15:50:08.179637 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170886    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 26 15:50:08.179637 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170890    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 26 15:50:08.180094 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170893    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 26 15:50:08.180094 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170896    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 26 15:50:08.180094 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170899    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 26 15:50:08.180094 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170901    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 26 15:50:08.180094 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170903    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 26 15:50:08.180094 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170906    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 26 15:50:08.180094 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170909    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 26 15:50:08.180094 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170912    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 26 15:50:08.180094 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170914    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 26 15:50:08.180094 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170917    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 26 15:50:08.180094 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170920    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 26 15:50:08.180094 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170922    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 26 15:50:08.180094 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170925    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 26 15:50:08.180094 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170928    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 26 15:50:08.180094 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170931    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 26 15:50:08.180094 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170934    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 26 15:50:08.180094 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170937    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 26 15:50:08.180094 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170939    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 26 15:50:08.180094 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170942    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 26 15:50:08.180094 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170944    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 26 15:50:08.180571 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170947    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 26 15:50:08.180571 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170949    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 26 15:50:08.180571 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170952    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 26 15:50:08.180571 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170954    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 26 15:50:08.180571 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170957    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 26 15:50:08.180571 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170959    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 26 15:50:08.180571 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170962    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 26 15:50:08.180571 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170964    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 26 15:50:08.180571 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170968    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 26 15:50:08.180571 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170971    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 26 15:50:08.180571 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170974    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 26 15:50:08.180571 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170977    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 26 15:50:08.180571 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170980    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 26 15:50:08.180571 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170983    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 26 15:50:08.180571 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170985    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 26 15:50:08.180571 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170988    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 26 15:50:08.180571 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170990    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 26 15:50:08.180571 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170993    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 26 15:50:08.180571 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170995    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 26 15:50:08.181052 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.170999    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 26 15:50:08.181052 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.171001    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 26 15:50:08.181052 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.171003    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 26 15:50:08.181052 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.171009    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 26 15:50:08.181052 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.177203    2563 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 26 15:50:08.181052 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.177217    2563 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 26 15:50:08.181052 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.177235    2563 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 26 15:50:08.181052 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.177242    2563 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 26 15:50:08.181052 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177315    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 26 15:50:08.181052 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177320    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 26 15:50:08.181052 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177323    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 26 15:50:08.181052 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177326    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 26 15:50:08.181052 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177329    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 26 15:50:08.181052 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177332    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 26 15:50:08.181052 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177335    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 26 15:50:08.181421 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177338    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 26 15:50:08.181421 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177341    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 26 15:50:08.181421 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177343    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 26 15:50:08.181421 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177346    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 26 15:50:08.181421 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177348    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 26 15:50:08.181421 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177350    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 26 15:50:08.181421 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177353    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 26 15:50:08.181421 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177355    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 26 15:50:08.181421 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177358    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 26 15:50:08.181421 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177360    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 26 15:50:08.181421 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177363    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 26 15:50:08.181421 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177365    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 26 15:50:08.181421 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177368    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 26 15:50:08.181421 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177371    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 26 15:50:08.181421 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177375    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 26 15:50:08.181421 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177378    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 26 15:50:08.181421 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177380    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 26 15:50:08.181421 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177383    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 26 15:50:08.181421 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177385    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 26 15:50:08.181875 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177388    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 26 15:50:08.181875 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177391    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 26 15:50:08.181875 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177394    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 26 15:50:08.181875 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177398    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 26 15:50:08.181875 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177400    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 26 15:50:08.181875 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177403    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 26 15:50:08.181875 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177405    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 26 15:50:08.181875 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177408    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 26 15:50:08.181875 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177410    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 26 15:50:08.181875 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177413    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 26 15:50:08.181875 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177415    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 26 15:50:08.181875 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177418    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 26 15:50:08.181875 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177420    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 26 15:50:08.181875 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177423    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 26 15:50:08.181875 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177425    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 26 15:50:08.181875 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177428    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 26 15:50:08.181875 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177431    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 26 15:50:08.181875 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177434    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 26 15:50:08.181875 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177436    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 26 15:50:08.181875 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177438    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 26 15:50:08.182343 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177441    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 26 15:50:08.182343 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177444    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 26 15:50:08.182343 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177446    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 26 15:50:08.182343 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177448    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 26 15:50:08.182343 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177451    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 26 15:50:08.182343 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177453    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 26 15:50:08.182343 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177456    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 26 15:50:08.182343 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177458    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 26 15:50:08.182343 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177461    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 26 15:50:08.182343 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177464    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 26 15:50:08.182343 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177466    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 26 15:50:08.182343 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177469    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 26 15:50:08.182343 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177472    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 26 15:50:08.182343 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177474    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 26 15:50:08.182343 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177477    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 26 15:50:08.182343 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177479    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 26 15:50:08.182343 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177482    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 26 15:50:08.182343 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177484    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 26 15:50:08.182343 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177487    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 26 15:50:08.182343 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177490    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 26 15:50:08.182892 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177492    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 26 15:50:08.182892 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177494    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 26 15:50:08.182892 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177497    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 26 15:50:08.182892 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177499    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 26 15:50:08.182892 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177501    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 26 15:50:08.182892 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177504    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 26 15:50:08.182892 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177506    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 26 15:50:08.182892 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177509    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 26 15:50:08.182892 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177513    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 26 15:50:08.182892 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177516    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 26 15:50:08.182892 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177519    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 26 15:50:08.182892 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177522    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 26 15:50:08.182892 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177525    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 26 15:50:08.182892 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177528    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 26 15:50:08.182892 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177532    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 26 15:50:08.182892 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177535    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 26 15:50:08.182892 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177537    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 26 15:50:08.182892 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177540    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 26 15:50:08.182892 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177542    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 26 15:50:08.183419 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177544    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 26 15:50:08.183419 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177547    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 26 15:50:08.183419 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177549    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 26 15:50:08.183419 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177552    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 26 15:50:08.183419 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177554    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 26 15:50:08.183419 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177557    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 26 15:50:08.183419 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177559    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 26 15:50:08.183419 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177562    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 26 15:50:08.183419 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177564    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 26 15:50:08.183419 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177567    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 26 15:50:08.183419 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177569    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 26 15:50:08.183419 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177572    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 26 15:50:08.183419 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177574    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 26 15:50:08.183419 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177577    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 26 15:50:08.183419 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.177581    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 26 15:50:08.184013 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177695    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 26 15:50:08.184013 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177700    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 26 15:50:08.184013 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177703    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 26 15:50:08.184013 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177706    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 26 15:50:08.184013 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177709    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 26 15:50:08.184013 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177712    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 26 15:50:08.184013 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177715    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 26 15:50:08.184013 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177718    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 26 15:50:08.184013 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177720    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 26 15:50:08.184013 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177723    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 26 15:50:08.184013 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177725    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 26 15:50:08.184013 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177729    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 26 15:50:08.184013 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177732    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 26 15:50:08.184013 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177735    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 26 15:50:08.184013 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177738    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 26 15:50:08.184013 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177741    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 26 15:50:08.184013 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177743    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 26 15:50:08.184013 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177746    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 26 15:50:08.184013 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177749    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 26 15:50:08.184704 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177751    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 26 15:50:08.184704 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177754    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 26 15:50:08.184704 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177756    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 26 15:50:08.184704 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177772    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 26 15:50:08.184704 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177775    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 26 15:50:08.184704 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177778    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 26 15:50:08.184704 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177780    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 26 15:50:08.184704 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177783    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 26 15:50:08.184704 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177786    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 26 15:50:08.184704 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177790    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 26 15:50:08.184704 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177793    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 26 15:50:08.184704 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177795    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 26 15:50:08.184704 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177798    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 26 15:50:08.184704 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177800    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 26 15:50:08.184704 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177802    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 26 15:50:08.184704 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177805    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 26 15:50:08.184704 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177807    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 26 15:50:08.184704 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177810    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 26 15:50:08.184704 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177812    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 26 15:50:08.185172 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177814    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 26 15:50:08.185172 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177817    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 26 15:50:08.185172 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177819    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 26 15:50:08.185172 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177822    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 26 15:50:08.185172 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177824    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 26 15:50:08.185172 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177827    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 26 15:50:08.185172 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177829    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 26 15:50:08.185172 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177832    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 26 15:50:08.185172 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177834    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 26 15:50:08.185172 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177836    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 26 15:50:08.185172 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177839    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 26 15:50:08.185172 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177842    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 26 15:50:08.185172 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177844    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 26 15:50:08.185172 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177846    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 26 15:50:08.185172 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177849    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 26 15:50:08.185172 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177851    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 26 15:50:08.185172 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177854    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 26 15:50:08.185172 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177856    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 26 15:50:08.185172 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177858    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 26 15:50:08.185172 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177861    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 26 15:50:08.185665 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177863    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 26 15:50:08.185665 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177866    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 26 15:50:08.185665 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177868    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 26 15:50:08.185665 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177872    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 26 15:50:08.185665 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177874    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 26 15:50:08.185665 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177877    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 26 15:50:08.185665 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177879    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 26 15:50:08.185665 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177881    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 26 15:50:08.185665 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177884    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 26 15:50:08.185665 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177886    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 26 15:50:08.185665 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177888    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 26 15:50:08.185665 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177891    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 26 15:50:08.185665 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177893    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 26 15:50:08.185665 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177895    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 26 15:50:08.185665 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177898    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 26 15:50:08.185665 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177900    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 26 15:50:08.185665 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177902    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 26 15:50:08.185665 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177905    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 26 15:50:08.185665 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177907    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 26 15:50:08.185665 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177910    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 26 15:50:08.186372 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177912    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 26 15:50:08.186372 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177914    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 26 15:50:08.186372 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177917    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 26 15:50:08.186372 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177920    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 26 15:50:08.186372 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177922    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 26 15:50:08.186372 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177924    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 26 15:50:08.186372 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177927    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 26 15:50:08.186372 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177929    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 26 15:50:08.186372 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177931    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 26 15:50:08.186372 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177934    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 26 15:50:08.186372 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177937    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 26 15:50:08.186372 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177939    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 26 15:50:08.186372 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177941    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 26 15:50:08.186372 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177944    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 26 15:50:08.186372 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177946    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 26 15:50:08.186372 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177948    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 26 15:50:08.186372 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177954    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 26 15:50:08.186372 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177957    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 26 15:50:08.186372 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177959    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 26 15:50:08.186841 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177961    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 26 15:50:08.186841 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:08.177964    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 26 15:50:08.186841 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.177969    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 26 15:50:08.186841 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.178064    2563 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 26 15:50:08.186841 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.181186    2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 26 15:50:08.186841 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.182400    2563 server.go:1019] "Starting client certificate rotation"
Jul 26 15:50:08.186841 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.182495    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 26 15:50:08.186841 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.183093    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 26 15:50:08.186841 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.186385    2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 26 15:50:08.188336 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.188319    2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 26 15:50:08.198783 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.198755    2563 log.go:25] "Validated CRI v1 runtime API"
Jul 26 15:50:08.200741 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.200726    2563 log.go:25] "Validated CRI v1 image API"
Jul 26 15:50:08.201492 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.201477    2563 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 26 15:50:08.204334 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.204316    2563 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 858678a3-4368-4eb7-bff3-8214273e7137:/dev/nvme0n1p3 d5836036-6701-4f5c-9083-876d5203c6b4:/dev/nvme0n1p4]
Jul 26 15:50:08.204400 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.204334    2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 26 15:50:08.210312 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.210202    2563 manager.go:216] Machine: {Timestamp:2026-07-26 15:50:08.208339245 +0000 UTC m=+0.312565045 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3204533 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f47280bdfe6c8d54b41f5d7833061 SystemUUID:ec2f4728-0bdf-e6c8-d54b-41f5d7833061 BootID:acd2f839-e826-4830-9182-bcd2e8368692 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:a8:34:61:31:f5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:a8:34:61:31:f5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:2a:52:49:20:53:17 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 26 15:50:08.210312 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.210304    2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 26 15:50:08.210443 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.210384    2563 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 26 15:50:08.210736 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.210719    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 26 15:50:08.213113 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.213085    2563 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 26 15:50:08.213258 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.213115    2563 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-204.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 26 15:50:08.213737 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.213729    2563 topology_manager.go:138] "Creating topology manager with none policy"
Jul 26 15:50:08.213785 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.213740    2563 container_manager_linux.go:309] "Creating device plugin manager"
Jul 26 15:50:08.213785 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.213747    2563 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 26 15:50:08.214522 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.214512    2563 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 26 15:50:08.215349 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.215340    2563 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 26 15:50:08.217983 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.217972    2563 state_mem.go:36] "Initialized new in-memory state store"
Jul 26 15:50:08.218087 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.218078    2563 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 26 15:50:08.218122 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.218106    2563 kubelet.go:486] "Attempting to sync node with API server"
Jul 26 15:50:08.218122 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.218115    2563 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 26 15:50:08.218174 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.218129    2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 26 15:50:08.218174 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.218137    2563 kubelet.go:398] "Adding apiserver pod source"
Jul 26 15:50:08.218174 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.218145    2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 26 15:50:08.218629 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.218618    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 26 15:50:08.218663 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.218638    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 26 15:50:08.222871 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.222852    2563 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 26 15:50:08.224747 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.224732    2563 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 26 15:50:08.224805 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.224778    2563 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 26 15:50:08.226812 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.226800    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 26 15:50:08.226812 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.226819    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 26 15:50:08.226812 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.226826    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 26 15:50:08.226812 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.226831    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 26 15:50:08.226952 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.226839    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 26 15:50:08.226952 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.226845    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 26 15:50:08.226952 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.226853    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 26 15:50:08.226952 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.226858    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 26 15:50:08.226952 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.226865    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 26 15:50:08.226952 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.226870    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 26 15:50:08.226952 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.226879    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 26 15:50:08.226952 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.226884    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 26 15:50:08.227576 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.227565    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 26 15:50:08.227576 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.227575    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 26 15:50:08.229129 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.229112    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rx97h"
Jul 26 15:50:08.230849 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.230836    2563 server.go:1268] "Started kubelet"
Jul 26 15:50:08.230967 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.230922    2563 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 26 15:50:08.231108 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.230914    2563 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 26 15:50:08.231246 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:08.231224    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-204.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 26 15:50:08.231340 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.231247    2563 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 26 15:50:08.231340 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.231293    2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-204.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 26 15:50:08.231439 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:08.231402    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 26 15:50:08.231714 ip-10-0-133-204 systemd[1]: Started Kubernetes Kubelet.
Jul 26 15:50:08.232704 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.232607    2563 server.go:310] "Adding debug handlers to kubelet server"
Jul 26 15:50:08.232704 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.232677    2563 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 26 15:50:08.235830 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.235813    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rx97h"
Jul 26 15:50:08.236601 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:08.236584    2563 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 26 15:50:08.239495 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.239475    2563 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 26 15:50:08.239495 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.239484    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 26 15:50:08.239639 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:08.239583    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-204.ec2.internal\" not found"
Jul 26 15:50:08.240048 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.240031    2563 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 26 15:50:08.240048 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.240044    2563 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 26 15:50:08.240198 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.240058    2563 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 26 15:50:08.240248 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.240208    2563 reconstruct.go:97] "Volume reconstruction finished"
Jul 26 15:50:08.240248 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.240216    2563 reconciler.go:29] "Reconciler: start to sync state"
Jul 26 15:50:08.240409 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.240353    2563 factory.go:55] Registering systemd factory
Jul 26 15:50:08.240409 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.240370    2563 factory.go:223] Registration of the systemd container factory successfully
Jul 26 15:50:08.240532 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:08.239562    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-204.ec2.internal.18c5e1b81ee4cd6a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-204.ec2.internal,UID:ip-10-0-133-204.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-204.ec2.internal,},FirstTimestamp:2026-07-26 15:50:08.230813034 +0000 UTC m=+0.335038833,LastTimestamp:2026-07-26 15:50:08.230813034 +0000 UTC m=+0.335038833,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-204.ec2.internal,}"
Jul 26 15:50:08.242213 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.242195    2563 factory.go:153] Registering CRI-O factory
Jul 26 15:50:08.242213 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.242214    2563 factory.go:223] Registration of the crio container factory successfully
Jul 26 15:50:08.242340 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.242262    2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 26 15:50:08.242340 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.242285    2563 factory.go:103] Registering Raw factory
Jul 26 15:50:08.242340 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.242299    2563 manager.go:1196] Started watching for new ooms in manager
Jul 26 15:50:08.242863 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.242695    2563 manager.go:349] Starting recovery of all containers
Jul 26 15:50:08.250468 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.250445    2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 26 15:50:08.253675 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:08.253654    2563 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-133-204.ec2.internal\" not found" node="ip-10-0-133-204.ec2.internal"
Jul 26 15:50:08.255082 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.255067    2563 manager.go:354] Recovery completed
Jul 26 15:50:08.259175 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.259164    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 26 15:50:08.260866 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.260849    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-204.ec2.internal" event="NodeHasSufficientMemory"
Jul 26 15:50:08.260928 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.260879    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-204.ec2.internal" event="NodeHasNoDiskPressure"
Jul 26 15:50:08.260928 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.260890    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-204.ec2.internal" event="NodeHasSufficientPID"
Jul 26 15:50:08.261444 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.261432    2563 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 26 15:50:08.261444 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.261444    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 26 15:50:08.261546 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.261443    2563 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 26 15:50:08.261546 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.261480    2563 state_mem.go:36] "Initialized new in-memory state store"
Jul 26 15:50:08.262152 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.262138    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-204.ec2.internal" event="NodeHasSufficientMemory"
Jul 26 15:50:08.262205 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.262162    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-204.ec2.internal" event="NodeHasNoDiskPressure"
Jul 26 15:50:08.262205 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.262172    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-204.ec2.internal" event="NodeHasSufficientPID"
Jul 26 15:50:08.263899 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.263886    2563 policy_none.go:49] "None policy: Start"
Jul 26 15:50:08.263957 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.263904    2563 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 26 15:50:08.263957 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.263917    2563 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 26 15:50:08.265811 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.265799    2563 policy_none.go:47] "Start"
Jul 26 15:50:08.295302 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.295285    2563 manager.go:337] "Starting Device Plugin manager"
Jul 26 15:50:08.295388 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:08.295316    2563 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 26 15:50:08.295388 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.295324    2563 server.go:87] "Starting device plugin registration server"
Jul 26 15:50:08.295539 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.295528    2563 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 26 15:50:08.295587 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.295540    2563 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 26 15:50:08.295638 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.295624    2563 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 26 15:50:08.295696 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.295687    2563 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 26 15:50:08.295696 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.295696    2563 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 26 15:50:08.296267 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:08.296248    2563 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 26 15:50:08.296345 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:08.296301    2563 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-204.ec2.internal\" not found"
Jul 26 15:50:08.365374 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.365334    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 26 15:50:08.367436 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.366422    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 26 15:50:08.367436 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.366454    2563 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 26 15:50:08.367436 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.366476    2563 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 26 15:50:08.367436 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:08.366506    2563 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 26 15:50:08.369469 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.369452    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 26 15:50:08.396540 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.396523    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 26 15:50:08.397875 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.397859    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-204.ec2.internal" event="NodeHasSufficientMemory"
Jul 26 15:50:08.397961 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.397896    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-204.ec2.internal" event="NodeHasNoDiskPressure"
Jul 26 15:50:08.397961 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.397911    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-204.ec2.internal" event="NodeHasSufficientPID"
Jul 26 15:50:08.397961 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.397940    2563 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-204.ec2.internal"
Jul 26 15:50:08.406876 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.406857    2563 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-204.ec2.internal"
Jul 26 15:50:08.406946 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:08.406877    2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-204.ec2.internal\": node \"ip-10-0-133-204.ec2.internal\" not found"
Jul 26 15:50:08.423714 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:08.423662    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-204.ec2.internal\" not found"
Jul 26 15:50:08.467321 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.467284    2563 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-204.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-204.ec2.internal"]
Jul 26 15:50:08.467394 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.467355    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 26 15:50:08.468689 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.468675    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-204.ec2.internal" event="NodeHasSufficientMemory"
Jul 26 15:50:08.468785 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.468708    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-204.ec2.internal" event="NodeHasNoDiskPressure"
Jul 26 15:50:08.468785 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.468722    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-204.ec2.internal" event="NodeHasSufficientPID"
Jul 26 15:50:08.471047 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.471033    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 26 15:50:08.471150 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.471137    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-204.ec2.internal"
Jul 26 15:50:08.471192 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.471164    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 26 15:50:08.471776 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.471748    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-204.ec2.internal" event="NodeHasSufficientMemory"
Jul 26 15:50:08.471855 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.471789    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-204.ec2.internal" event="NodeHasNoDiskPressure"
Jul 26 15:50:08.471855 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.471799    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-204.ec2.internal" event="NodeHasSufficientPID"
Jul 26 15:50:08.471855 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.471750    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-204.ec2.internal" event="NodeHasSufficientMemory"
Jul 26 15:50:08.471954 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.471867    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-204.ec2.internal" event="NodeHasNoDiskPressure"
Jul 26 15:50:08.471954 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.471881    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-204.ec2.internal" event="NodeHasSufficientPID"
Jul 26 15:50:08.474231 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.474216    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-204.ec2.internal"
Jul 26 15:50:08.474293 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.474240    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 26 15:50:08.474895 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.474880    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-204.ec2.internal" event="NodeHasSufficientMemory"
Jul 26 15:50:08.474961 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.474912    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-204.ec2.internal" event="NodeHasNoDiskPressure"
Jul 26 15:50:08.474961 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.474927    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-204.ec2.internal" event="NodeHasSufficientPID"
Jul 26 15:50:08.491241 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:08.491225    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-204.ec2.internal\" not found" node="ip-10-0-133-204.ec2.internal"
Jul 26 15:50:08.495016 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:08.495002    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-204.ec2.internal\" not found" node="ip-10-0-133-204.ec2.internal"
Jul 26 15:50:08.524161 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:08.524139    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-204.ec2.internal\" not found"
Jul 26 15:50:08.542429 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.542413    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6db9e0d726c4428a7173af2b853a8bd3-config\") pod \"kube-apiserver-proxy-ip-10-0-133-204.ec2.internal\" (UID: \"6db9e0d726c4428a7173af2b853a8bd3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-204.ec2.internal"
Jul 26 15:50:08.542509 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.542436    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5e83b5d2069ff1f9446be40c52eae0c3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-204.ec2.internal\" (UID: \"5e83b5d2069ff1f9446be40c52eae0c3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-204.ec2.internal"
Jul 26 15:50:08.542509 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.542457    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e83b5d2069ff1f9446be40c52eae0c3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-204.ec2.internal\" (UID: \"5e83b5d2069ff1f9446be40c52eae0c3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-204.ec2.internal"
Jul 26 15:50:08.624661 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:08.624634    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-204.ec2.internal\" not found"
Jul 26 15:50:08.642826 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.642631    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6db9e0d726c4428a7173af2b853a8bd3-config\") pod \"kube-apiserver-proxy-ip-10-0-133-204.ec2.internal\" (UID: \"6db9e0d726c4428a7173af2b853a8bd3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-204.ec2.internal"
Jul 26 15:50:08.642896 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.642838    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5e83b5d2069ff1f9446be40c52eae0c3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-204.ec2.internal\" (UID: \"5e83b5d2069ff1f9446be40c52eae0c3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-204.ec2.internal"
Jul 26 15:50:08.642896 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.642858    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e83b5d2069ff1f9446be40c52eae0c3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-204.ec2.internal\" (UID: \"5e83b5d2069ff1f9446be40c52eae0c3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-204.ec2.internal"
Jul 26 15:50:08.642976 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.642733    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6db9e0d726c4428a7173af2b853a8bd3-config\") pod \"kube-apiserver-proxy-ip-10-0-133-204.ec2.internal\" (UID: \"6db9e0d726c4428a7173af2b853a8bd3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-204.ec2.internal"
Jul 26 15:50:08.642976 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.642932    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e83b5d2069ff1f9446be40c52eae0c3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-204.ec2.internal\" (UID: \"5e83b5d2069ff1f9446be40c52eae0c3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-204.ec2.internal"
Jul 26 15:50:08.642976 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.642941    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5e83b5d2069ff1f9446be40c52eae0c3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-204.ec2.internal\" (UID: \"5e83b5d2069ff1f9446be40c52eae0c3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-204.ec2.internal"
Jul 26 15:50:08.725286 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:08.725204    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-204.ec2.internal\" not found"
Jul 26 15:50:08.797119 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.797073    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-204.ec2.internal"
Jul 26 15:50:08.798240 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:08.798225    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-204.ec2.internal"
Jul 26 15:50:08.826207 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:08.826182    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-204.ec2.internal\" not found"
Jul 26 15:50:08.932361 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:08.932317    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-204.ec2.internal\" not found"
Jul 26 15:50:09.035018 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:09.034946    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-204.ec2.internal\" not found"
Jul 26 15:50:09.066149 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.066120    2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 26 15:50:09.098190 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.098157    2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 26 15:50:09.140653 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.140624    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-204.ec2.internal"
Jul 26 15:50:09.154016 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.153990    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 26 15:50:09.155043 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.155026    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-204.ec2.internal"
Jul 26 15:50:09.166799 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.166778    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 26 15:50:09.182500 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.182474    2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 26 15:50:09.182655 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.182624    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 26 15:50:09.182756 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.182666    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 26 15:50:09.182756 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.182678    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 26 15:50:09.182756 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.182665    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 26 15:50:09.218411 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.218374    2563 apiserver.go:52] "Watching apiserver"
Jul 26 15:50:09.233672 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.233643    2563 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 26 15:50:09.234800 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.234764    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-kwjfd","kube-system/konnectivity-agent-pj48f","kube-system/kube-apiserver-proxy-ip-10-0-133-204.ec2.internal","openshift-cluster-node-tuning-operator/tuned-pwcmg","openshift-image-registry/node-ca-p8kcf","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-204.ec2.internal","openshift-multus/network-metrics-daemon-8gzcl","kube-system/global-pull-secret-syncer-cxfp4","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zvsxm","openshift-dns/node-resolver-gnkxb","openshift-multus/multus-additional-cni-plugins-plqcw","openshift-multus/multus-t7zjp","openshift-network-diagnostics/network-check-target-7g75t","openshift-network-operator/iptables-alerter-jltnr"]
Jul 26 15:50:09.237586 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.237560    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-p8kcf"
Jul 26 15:50:09.239821 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.239796    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 26 15:50:09.240067 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.240038    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-k9rbg\""
Jul 26 15:50:09.240067 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.240060    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 26 15:50:09.240170 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.240039    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 26 15:50:09.240361 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.240339    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 26 15:50:09.240550 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.240534    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-plqcw"
Jul 26 15:50:09.240869 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.240772    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-25 15:45:08 +0000 UTC" deadline="2028-04-11 20:10:46.362992774 +0000 UTC"
Jul 26 15:50:09.240957 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.240868    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15004h20m37.122128749s"
Jul 26 15:50:09.242721 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.242703    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 26 15:50:09.242799 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.242703    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 26 15:50:09.242960 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.242945    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-kvcwl\""
Jul 26 15:50:09.243021 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.242998    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 26 15:50:09.243076 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.243026    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 26 15:50:09.243224 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.243210    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 26 15:50:09.243351 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.243316    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kwjfd"
Jul 26 15:50:09.245487 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.245468    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 26 15:50:09.245586 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.245469    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-s75px\""
Jul 26 15:50:09.245586 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.245522    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 26 15:50:09.245966 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.245947    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 26 15:50:09.246042 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.245957    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 26 15:50:09.246120 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.246104    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 26 15:50:09.246878 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.246633    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8gzcl"
Jul 26 15:50:09.246878 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:09.246752    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8gzcl" podUID="ca110f78-ba25-4be3-9bed-60e5ef15e2c5"
Jul 26 15:50:09.247244 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.247160    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 26 15:50:09.247302 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.247230    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a415587b-d9d6-4f2d-a37b-a49bb34aba81-cnibin\") pod \"multus-additional-cni-plugins-plqcw\" (UID: \"a415587b-d9d6-4f2d-a37b-a49bb34aba81\") " pod="openshift-multus/multus-additional-cni-plugins-plqcw"
Jul 26 15:50:09.247302 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.247284    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a415587b-d9d6-4f2d-a37b-a49bb34aba81-tuning-conf-dir\") pod \"multus-additional-cni-plugins-plqcw\" (UID: \"a415587b-d9d6-4f2d-a37b-a49bb34aba81\") " pod="openshift-multus/multus-additional-cni-plugins-plqcw"
Jul 26 15:50:09.247407 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.247324    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a415587b-d9d6-4f2d-a37b-a49bb34aba81-os-release\") pod \"multus-additional-cni-plugins-plqcw\" (UID: \"a415587b-d9d6-4f2d-a37b-a49bb34aba81\") " pod="openshift-multus/multus-additional-cni-plugins-plqcw"
Jul 26 15:50:09.247407 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.247366    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a415587b-d9d6-4f2d-a37b-a49bb34aba81-cni-binary-copy\") pod \"multus-additional-cni-plugins-plqcw\" (UID: \"a415587b-d9d6-4f2d-a37b-a49bb34aba81\") " pod="openshift-multus/multus-additional-cni-plugins-plqcw"
Jul 26 15:50:09.247510 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.247405    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a415587b-d9d6-4f2d-a37b-a49bb34aba81-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-plqcw\" (UID: \"a415587b-d9d6-4f2d-a37b-a49bb34aba81\") " pod="openshift-multus/multus-additional-cni-plugins-plqcw"
Jul 26 15:50:09.247510 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.247441    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a415587b-d9d6-4f2d-a37b-a49bb34aba81-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-plqcw\" (UID: \"a415587b-d9d6-4f2d-a37b-a49bb34aba81\") " pod="openshift-multus/multus-additional-cni-plugins-plqcw"
Jul 26 15:50:09.247510 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.247479    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5lc8\" (UniqueName: \"kubernetes.io/projected/a415587b-d9d6-4f2d-a37b-a49bb34aba81-kube-api-access-t5lc8\") pod \"multus-additional-cni-plugins-plqcw\" (UID: \"a415587b-d9d6-4f2d-a37b-a49bb34aba81\") " pod="openshift-multus/multus-additional-cni-plugins-plqcw"
Jul 26 15:50:09.247657 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.247525    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e1f2fc7a-9737-422c-9c17-c8821dc48396-host\") pod \"node-ca-p8kcf\" (UID: \"e1f2fc7a-9737-422c-9c17-c8821dc48396\") " pod="openshift-image-registry/node-ca-p8kcf"
Jul 26 15:50:09.247657 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.247563    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e1f2fc7a-9737-422c-9c17-c8821dc48396-serviceca\") pod \"node-ca-p8kcf\" (UID: \"e1f2fc7a-9737-422c-9c17-c8821dc48396\") " pod="openshift-image-registry/node-ca-p8kcf"
Jul 26 15:50:09.247657 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.247598    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ms68\" (UniqueName: \"kubernetes.io/projected/e1f2fc7a-9737-422c-9c17-c8821dc48396-kube-api-access-8ms68\") pod \"node-ca-p8kcf\" (UID: \"e1f2fc7a-9737-422c-9c17-c8821dc48396\") " pod="openshift-image-registry/node-ca-p8kcf"
Jul 26 15:50:09.247809 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.247669    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a415587b-d9d6-4f2d-a37b-a49bb34aba81-system-cni-dir\") pod \"multus-additional-cni-plugins-plqcw\" (UID: \"a415587b-d9d6-4f2d-a37b-a49bb34aba81\") " pod="openshift-multus/multus-additional-cni-plugins-plqcw"
Jul 26 15:50:09.250623 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.250603    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 26 15:50:09.250970 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.250955    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pwcmg"
Jul 26 15:50:09.253155 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.253138    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-dj7bg\""
Jul 26 15:50:09.253414 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.253395    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 26 15:50:09.253475 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.253435    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 26 15:50:09.253839 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.253822    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cxfp4"
Jul 26 15:50:09.253916 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:09.253896    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cxfp4" podUID="e62cc557-6267-4993-84a1-a888ced92345"
Jul 26 15:50:09.256853 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.256829    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zvsxm"
Jul 26 15:50:09.259165 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.259150    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 26 15:50:09.259405 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.259389    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-qpw4p\""
Jul 26 15:50:09.259486 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.259395    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 26 15:50:09.259554 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.259422    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 26 15:50:09.259862 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.259846    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gnkxb"
Jul 26 15:50:09.261619 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.261600    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 26 15:50:09.262027 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.262014    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 26 15:50:09.262118 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.262099    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-4qcfn\""
Jul 26 15:50:09.262726 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.262710    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pj48f"
Jul 26 15:50:09.264875 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.264859    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 26 15:50:09.264969 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.264948    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-npbfs\""
Jul 26 15:50:09.264969 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.264951    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 26 15:50:09.265620 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.265604    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-t7zjp"
Jul 26 15:50:09.267365 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.267351    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 26 15:50:09.267776 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.267761    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-225vd\""
Jul 26 15:50:09.268908 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.268895    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7g75t"
Jul 26 15:50:09.268967 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:09.268951    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7g75t" podUID="0c90caab-1908-4071-a259-e0343960cbe2"
Jul 26 15:50:09.271994 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.271718    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jltnr"
Jul 26 15:50:09.272568 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.272551    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-65n57"
Jul 26 15:50:09.274375 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.274327    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9j6f4\""
Jul 26 15:50:09.274375 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.274338    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 26 15:50:09.274375 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.274359    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 26 15:50:09.274730 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.274708    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 26 15:50:09.282533 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.282514    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-65n57"
Jul 26 15:50:09.341130 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.341105    2563 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 26 15:50:09.347891 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.347871    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ms68\" (UniqueName: \"kubernetes.io/projected/e1f2fc7a-9737-422c-9c17-c8821dc48396-kube-api-access-8ms68\") pod \"node-ca-p8kcf\" (UID: \"e1f2fc7a-9737-422c-9c17-c8821dc48396\") " pod="openshift-image-registry/node-ca-p8kcf"
Jul 26 15:50:09.347991 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.347909    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/31754f05-c0cf-4fa6-a3b5-f1afff0b7c99-env-overrides\") pod \"ovnkube-node-kwjfd\" (UID: \"31754f05-c0cf-4fa6-a3b5-f1afff0b7c99\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwjfd"
Jul 26 15:50:09.347991 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.347939    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e62cc557-6267-4993-84a1-a888ced92345-original-pull-secret\") pod \"global-pull-secret-syncer-cxfp4\" (UID: \"e62cc557-6267-4993-84a1-a888ced92345\") " pod="kube-system/global-pull-secret-syncer-cxfp4"
Jul 26 15:50:09.348109 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.347988    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a6f459fa-098f-40e8-9086-bde5014c0807-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zvsxm\" (UID: \"a6f459fa-098f-40e8-9086-bde5014c0807\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zvsxm"
Jul 26 15:50:09.348109 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.348017    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a6f459fa-098f-40e8-9086-bde5014c0807-socket-dir\") pod \"aws-ebs-csi-driver-node-zvsxm\" (UID: \"a6f459fa-098f-40e8-9086-bde5014c0807\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zvsxm"
Jul 26 15:50:09.348109 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.348043    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/705b8a13-4555-4820-8e91-e2ee6dbac16e-cnibin\") pod \"multus-t7zjp\" (UID: \"705b8a13-4555-4820-8e91-e2ee6dbac16e\") " pod="openshift-multus/multus-t7zjp"
Jul 26 15:50:09.348109 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.348082    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/705b8a13-4555-4820-8e91-e2ee6dbac16e-host-var-lib-kubelet\") pod \"multus-t7zjp\" (UID: \"705b8a13-4555-4820-8e91-e2ee6dbac16e\") " pod="openshift-multus/multus-t7zjp"
Jul 26 15:50:09.348281 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.348111    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a415587b-d9d6-4f2d-a37b-a49bb34aba81-os-release\") pod \"multus-additional-cni-plugins-plqcw\" (UID: \"a415587b-d9d6-4f2d-a37b-a49bb34aba81\") " pod="openshift-multus/multus-additional-cni-plugins-plqcw"
Jul 26 15:50:09.348281 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.348138    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/31754f05-c0cf-4fa6-a3b5-f1afff0b7c99-host-kubelet\") pod \"ovnkube-node-kwjfd\" (UID: \"31754f05-c0cf-4fa6-a3b5-f1afff0b7c99\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwjfd"
Jul 26 15:50:09.348281 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.348178    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/31754f05-c0cf-4fa6-a3b5-f1afff0b7c99-host-run-netns\") pod \"ovnkube-node-kwjfd\" (UID: \"31754f05-c0cf-4fa6-a3b5-f1afff0b7c99\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwjfd"
Jul 26 15:50:09.348281 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.348213    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/31754f05-c0cf-4fa6-a3b5-f1afff0b7c99-ovnkube-config\") pod \"ovnkube-node-kwjfd\" (UID: \"31754f05-c0cf-4fa6-a3b5-f1afff0b7c99\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwjfd"
Jul 26 15:50:09.348281 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.348236    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a415587b-d9d6-4f2d-a37b-a49bb34aba81-os-release\") pod \"multus-additional-cni-plugins-plqcw\" (UID: \"a415587b-d9d6-4f2d-a37b-a49bb34aba81\") " pod="openshift-multus/multus-additional-cni-plugins-plqcw"
Jul 26 15:50:09.348281 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.348237    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/705b8a13-4555-4820-8e91-e2ee6dbac16e-host-run-multus-certs\") pod \"multus-t7zjp\" (UID: \"705b8a13-4555-4820-8e91-e2ee6dbac16e\") " pod="openshift-multus/multus-t7zjp"
Jul 26 15:50:09.348549 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.348286    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a415587b-d9d6-4f2d-a37b-a49bb34aba81-cnibin\") pod \"multus-additional-cni-plugins-plqcw\" (UID: \"a415587b-d9d6-4f2d-a37b-a49bb34aba81\") " pod="openshift-multus/multus-additional-cni-plugins-plqcw"
Jul 26 15:50:09.348549 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.348304    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/31754f05-c0cf-4fa6-a3b5-f1afff0b7c99-var-lib-openvswitch\") pod \"ovnkube-node-kwjfd\" (UID: \"31754f05-c0cf-4fa6-a3b5-f1afff0b7c99\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwjfd"
Jul 26 15:50:09.348549 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.348327    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a59a089c-67e4-455b-9936-af62b889eda9-run\") pod \"tuned-pwcmg\" (UID: \"a59a089c-67e4-455b-9936-af62b889eda9\") " pod="openshift-cluster-node-tuning-operator/tuned-pwcmg"
Jul 26 15:50:09.348549 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.348352    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a415587b-d9d6-4f2d-a37b-a49bb34aba81-cnibin\") pod \"multus-additional-cni-plugins-plqcw\" (UID: \"a415587b-d9d6-4f2d-a37b-a49bb34aba81\") " pod="openshift-multus/multus-additional-cni-plugins-plqcw"
Jul 26 15:50:09.348549 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.348361    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a59a089c-67e4-455b-9936-af62b889eda9-lib-modules\") pod \"tuned-pwcmg\" (UID: \"a59a089c-67e4-455b-9936-af62b889eda9\") " pod="openshift-cluster-node-tuning-operator/tuned-pwcmg"
Jul 26 15:50:09.348549 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.348395    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a59a089c-67e4-455b-9936-af62b889eda9-etc-tuned\") pod \"tuned-pwcmg\" (UID: \"a59a089c-67e4-455b-9936-af62b889eda9\") " pod="openshift-cluster-node-tuning-operator/tuned-pwcmg"
Jul 26 15:50:09.348549 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.348420    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a6f459fa-098f-40e8-9086-bde5014c0807-device-dir\") pod \"aws-ebs-csi-driver-node-zvsxm\" (UID: \"a6f459fa-098f-40e8-9086-bde5014c0807\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zvsxm"
Jul 26 15:50:09.348549 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.348445    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt8pd\" (UniqueName: \"kubernetes.io/projected/236e97e8-5df0-4a36-9513-a260891d2e9e-kube-api-access-qt8pd\") pod \"iptables-alerter-jltnr\" (UID: \"236e97e8-5df0-4a36-9513-a260891d2e9e\") " pod="openshift-network-operator/iptables-alerter-jltnr"
Jul 26 15:50:09.348549 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.348479    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/31754f05-c0cf-4fa6-a3b5-f1afff0b7c99-node-log\") pod \"ovnkube-node-kwjfd\" (UID: \"31754f05-c0cf-4fa6-a3b5-f1afff0b7c99\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwjfd"
Jul 26 15:50:09.348549 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.348518    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e62cc557-6267-4993-84a1-a888ced92345-dbus\") pod \"global-pull-secret-syncer-cxfp4\" (UID: \"e62cc557-6267-4993-84a1-a888ced92345\") " pod="kube-system/global-pull-secret-syncer-cxfp4"
Jul 26 15:50:09.348549 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.348548    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/aba3d036-e332-4b06-8379-7deb395345ba-hosts-file\") pod \"node-resolver-gnkxb\" (UID: \"aba3d036-e332-4b06-8379-7deb395345ba\") " pod="openshift-dns/node-resolver-gnkxb"
Jul 26 15:50:09.348978 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.348564    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/705b8a13-4555-4820-8e91-e2ee6dbac16e-host-run-k8s-cni-cncf-io\") pod \"multus-t7zjp\" (UID: \"705b8a13-4555-4820-8e91-e2ee6dbac16e\") " pod="openshift-multus/multus-t7zjp"
Jul 26 15:50:09.348978 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.348583    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/236e97e8-5df0-4a36-9513-a260891d2e9e-iptables-alerter-script\") pod \"iptables-alerter-jltnr\" (UID: \"236e97e8-5df0-4a36-9513-a260891d2e9e\") " pod="openshift-network-operator/iptables-alerter-jltnr"
Jul 26 15:50:09.348978 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.348597    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/236e97e8-5df0-4a36-9513-a260891d2e9e-host-slash\") pod \"iptables-alerter-jltnr\" (UID: \"236e97e8-5df0-4a36-9513-a260891d2e9e\") " pod="openshift-network-operator/iptables-alerter-jltnr"
Jul 26 15:50:09.348978 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.348616    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a415587b-d9d6-4f2d-a37b-a49bb34aba81-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-plqcw\" (UID: \"a415587b-d9d6-4f2d-a37b-a49bb34aba81\") " pod="openshift-multus/multus-additional-cni-plugins-plqcw"
Jul 26 15:50:09.348978 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.348651    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6rm4\" (UniqueName: \"kubernetes.io/projected/31754f05-c0cf-4fa6-a3b5-f1afff0b7c99-kube-api-access-b6rm4\") pod \"ovnkube-node-kwjfd\" (UID: \"31754f05-c0cf-4fa6-a3b5-f1afff0b7c99\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwjfd"
Jul 26 15:50:09.348978 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.348733    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a59a089c-67e4-455b-9936-af62b889eda9-etc-systemd\") pod \"tuned-pwcmg\" (UID: \"a59a089c-67e4-455b-9936-af62b889eda9\") " pod="openshift-cluster-node-tuning-operator/tuned-pwcmg"
Jul 26 15:50:09.348978 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.348758    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a6f459fa-098f-40e8-9086-bde5014c0807-sys-fs\") pod \"aws-ebs-csi-driver-node-zvsxm\" (UID: \"a6f459fa-098f-40e8-9086-bde5014c0807\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zvsxm"
Jul 26 15:50:09.348978 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.348833    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aba3d036-e332-4b06-8379-7deb395345ba-tmp-dir\") pod \"node-resolver-gnkxb\" (UID: \"aba3d036-e332-4b06-8379-7deb395345ba\") " pod="openshift-dns/node-resolver-gnkxb"
Jul 26 15:50:09.348978 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.348879    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/705b8a13-4555-4820-8e91-e2ee6dbac16e-system-cni-dir\") pod \"multus-t7zjp\" (UID: \"705b8a13-4555-4820-8e91-e2ee6dbac16e\") " pod="openshift-multus/multus-t7zjp"
Jul 26 15:50:09.348978 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.348903    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/31754f05-c0cf-4fa6-a3b5-f1afff0b7c99-run-openvswitch\") pod \"ovnkube-node-kwjfd\" (UID: \"31754f05-c0cf-4fa6-a3b5-f1afff0b7c99\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwjfd"
Jul 26 15:50:09.348978 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.348926    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a59a089c-67e4-455b-9936-af62b889eda9-etc-sysctl-conf\") pod \"tuned-pwcmg\" (UID: \"a59a089c-67e4-455b-9936-af62b889eda9\") " pod="openshift-cluster-node-tuning-operator/tuned-pwcmg"
Jul 26 15:50:09.348978 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.348950    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/705b8a13-4555-4820-8e91-e2ee6dbac16e-host-var-lib-cni-bin\") pod \"multus-t7zjp\" (UID: \"705b8a13-4555-4820-8e91-e2ee6dbac16e\") " pod="openshift-multus/multus-t7zjp"
Jul 26 15:50:09.348978 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.348980    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a415587b-d9d6-4f2d-a37b-a49bb34aba81-system-cni-dir\") pod \"multus-additional-cni-plugins-plqcw\" (UID: \"a415587b-d9d6-4f2d-a37b-a49bb34aba81\") " pod="openshift-multus/multus-additional-cni-plugins-plqcw"
Jul 26 15:50:09.349509 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.349009    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca110f78-ba25-4be3-9bed-60e5ef15e2c5-metrics-certs\") pod \"network-metrics-daemon-8gzcl\" (UID: \"ca110f78-ba25-4be3-9bed-60e5ef15e2c5\") " pod="openshift-multus/network-metrics-daemon-8gzcl"
Jul 26 15:50:09.349509 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.349034    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a59a089c-67e4-455b-9936-af62b889eda9-tmp\") pod \"tuned-pwcmg\" (UID: \"a59a089c-67e4-455b-9936-af62b889eda9\") " pod="openshift-cluster-node-tuning-operator/tuned-pwcmg"
Jul 26 15:50:09.349509 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.349059    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a6f459fa-098f-40e8-9086-bde5014c0807-registration-dir\") pod \"aws-ebs-csi-driver-node-zvsxm\" (UID: \"a6f459fa-098f-40e8-9086-bde5014c0807\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zvsxm"
Jul 26 15:50:09.349509 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.349066    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a415587b-d9d6-4f2d-a37b-a49bb34aba81-system-cni-dir\") pod \"multus-additional-cni-plugins-plqcw\" (UID: \"a415587b-d9d6-4f2d-a37b-a49bb34aba81\") " pod="openshift-multus/multus-additional-cni-plugins-plqcw"
Jul 26 15:50:09.349509 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.349091    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/705b8a13-4555-4820-8e91-e2ee6dbac16e-multus-daemon-config\") pod \"multus-t7zjp\" (UID: \"705b8a13-4555-4820-8e91-e2ee6dbac16e\") " pod="openshift-multus/multus-t7zjp"
Jul 26 15:50:09.349509 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.349136    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/31754f05-c0cf-4fa6-a3b5-f1afff0b7c99-host-slash\") pod \"ovnkube-node-kwjfd\" (UID: \"31754f05-c0cf-4fa6-a3b5-f1afff0b7c99\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwjfd"
Jul 26 15:50:09.349509 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.349156    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a415587b-d9d6-4f2d-a37b-a49bb34aba81-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-plqcw\" (UID: \"a415587b-d9d6-4f2d-a37b-a49bb34aba81\") " pod="openshift-multus/multus-additional-cni-plugins-plqcw"
Jul 26 15:50:09.349509 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.349166    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a59a089c-67e4-455b-9936-af62b889eda9-etc-sysctl-d\") pod \"tuned-pwcmg\" (UID: \"a59a089c-67e4-455b-9936-af62b889eda9\") " pod="openshift-cluster-node-tuning-operator/tuned-pwcmg"
Jul 26 15:50:09.349509 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.349199    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8zd9\" (UniqueName: \"kubernetes.io/projected/a59a089c-67e4-455b-9936-af62b889eda9-kube-api-access-k8zd9\") pod \"tuned-pwcmg\" (UID: \"a59a089c-67e4-455b-9936-af62b889eda9\") " pod="openshift-cluster-node-tuning-operator/tuned-pwcmg"
Jul 26 15:50:09.349509 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.349218    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwnhx\" (UniqueName: \"kubernetes.io/projected/aba3d036-e332-4b06-8379-7deb395345ba-kube-api-access-wwnhx\") pod \"node-resolver-gnkxb\" (UID: \"aba3d036-e332-4b06-8379-7deb395345ba\") " pod="openshift-dns/node-resolver-gnkxb"
Jul 26 15:50:09.349509 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.349237    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a59a089c-67e4-455b-9936-af62b889eda9-etc-modprobe-d\") pod \"tuned-pwcmg\" (UID: \"a59a089c-67e4-455b-9936-af62b889eda9\") " pod="openshift-cluster-node-tuning-operator/tuned-pwcmg"
Jul 26 15:50:09.349509 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.349260    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a59a089c-67e4-455b-9936-af62b889eda9-etc-sysconfig\") pod \"tuned-pwcmg\" (UID: \"a59a089c-67e4-455b-9936-af62b889eda9\") " pod="openshift-cluster-node-tuning-operator/tuned-pwcmg"
Jul 26 15:50:09.349509 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.349282    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a59a089c-67e4-455b-9936-af62b889eda9-sys\") pod \"tuned-pwcmg\" (UID: \"a59a089c-67e4-455b-9936-af62b889eda9\") " pod="openshift-cluster-node-tuning-operator/tuned-pwcmg"
Jul 26 15:50:09.349509 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.349308    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a59a089c-67e4-455b-9936-af62b889eda9-host\") pod \"tuned-pwcmg\" (UID: \"a59a089c-67e4-455b-9936-af62b889eda9\") " pod="openshift-cluster-node-tuning-operator/tuned-pwcmg"
Jul 26 15:50:09.349509 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.349322    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/705b8a13-4555-4820-8e91-e2ee6dbac16e-cni-binary-copy\") pod \"multus-t7zjp\" (UID: \"705b8a13-4555-4820-8e91-e2ee6dbac16e\") " pod="openshift-multus/multus-t7zjp"
Jul 26 15:50:09.349509 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.349347    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a415587b-d9d6-4f2d-a37b-a49bb34aba81-tuning-conf-dir\") pod \"multus-additional-cni-plugins-plqcw\" (UID: \"a415587b-d9d6-4f2d-a37b-a49bb34aba81\") " pod="openshift-multus/multus-additional-cni-plugins-plqcw"
Jul 26 15:50:09.350231 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.349370    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a415587b-d9d6-4f2d-a37b-a49bb34aba81-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-plqcw\" (UID: \"a415587b-d9d6-4f2d-a37b-a49bb34aba81\") " pod="openshift-multus/multus-additional-cni-plugins-plqcw"
Jul 26 15:50:09.350231 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.349397    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/31754f05-c0cf-4fa6-a3b5-f1afff0b7c99-etc-openvswitch\") pod \"ovnkube-node-kwjfd\" (UID: \"31754f05-c0cf-4fa6-a3b5-f1afff0b7c99\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwjfd"
Jul 26 15:50:09.350231 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.349421    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/31754f05-c0cf-4fa6-a3b5-f1afff0b7c99-run-ovn\") pod \"ovnkube-node-kwjfd\" (UID: \"31754f05-c0cf-4fa6-a3b5-f1afff0b7c99\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwjfd"
Jul 26 15:50:09.350231 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.349458    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/31754f05-c0cf-4fa6-a3b5-f1afff0b7c99-host-cni-bin\") pod \"ovnkube-node-kwjfd\" (UID: \"31754f05-c0cf-4fa6-a3b5-f1afff0b7c99\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwjfd"
Jul 26 15:50:09.350231 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.349497    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/31754f05-c0cf-4fa6-a3b5-f1afff0b7c99-host-cni-netd\") pod \"ovnkube-node-kwjfd\" (UID: \"31754f05-c0cf-4fa6-a3b5-f1afff0b7c99\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwjfd"
Jul 26 15:50:09.350231 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.349520    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a59a089c-67e4-455b-9936-af62b889eda9-etc-kubernetes\") pod \"tuned-pwcmg\" (UID: \"a59a089c-67e4-455b-9936-af62b889eda9\") " pod="openshift-cluster-node-tuning-operator/tuned-pwcmg"
Jul 26 15:50:09.350231 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.349539    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a415587b-d9d6-4f2d-a37b-a49bb34aba81-tuning-conf-dir\") pod \"multus-additional-cni-plugins-plqcw\" (UID: \"a415587b-d9d6-4f2d-a37b-a49bb34aba81\") " pod="openshift-multus/multus-additional-cni-plugins-plqcw"
Jul 26 15:50:09.350231 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.349544    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a59a089c-67e4-455b-9936-af62b889eda9-var-lib-kubelet\") pod \"tuned-pwcmg\" (UID: \"a59a089c-67e4-455b-9936-af62b889eda9\") " pod="openshift-cluster-node-tuning-operator/tuned-pwcmg"
Jul 26 15:50:09.350231 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.349591    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a415587b-d9d6-4f2d-a37b-a49bb34aba81-cni-binary-copy\") pod \"multus-additional-cni-plugins-plqcw\" (UID: \"a415587b-d9d6-4f2d-a37b-a49bb34aba81\") " pod="openshift-multus/multus-additional-cni-plugins-plqcw"
Jul 26 15:50:09.350231 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.349618    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/31754f05-c0cf-4fa6-a3b5-f1afff0b7c99-log-socket\") pod \"ovnkube-node-kwjfd\" (UID: \"31754f05-c0cf-4fa6-a3b5-f1afff0b7c99\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwjfd"
Jul 26 15:50:09.350231 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.349644    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/31754f05-c0cf-4fa6-a3b5-f1afff0b7c99-host-run-ovn-kubernetes\") pod \"ovnkube-node-kwjfd\" (UID: \"31754f05-c0cf-4fa6-a3b5-f1afff0b7c99\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwjfd"
Jul 26 15:50:09.350231 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.349685    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/31754f05-c0cf-4fa6-a3b5-f1afff0b7c99-ovnkube-script-lib\") pod \"ovnkube-node-kwjfd\" (UID: \"31754f05-c0cf-4fa6-a3b5-f1afff0b7c99\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwjfd"
Jul 26 15:50:09.350231 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.349714    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdsd2\" (UniqueName: \"kubernetes.io/projected/ca110f78-ba25-4be3-9bed-60e5ef15e2c5-kube-api-access-jdsd2\") pod \"network-metrics-daemon-8gzcl\" (UID: \"ca110f78-ba25-4be3-9bed-60e5ef15e2c5\") " pod="openshift-multus/network-metrics-daemon-8gzcl"
Jul 26 15:50:09.350231 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.349740    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a6f459fa-098f-40e8-9086-bde5014c0807-etc-selinux\") pod \"aws-ebs-csi-driver-node-zvsxm\" (UID: \"a6f459fa-098f-40e8-9086-bde5014c0807\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zvsxm"
Jul 26 15:50:09.350231 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.349773    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdpxp\" (UniqueName: \"kubernetes.io/projected/a6f459fa-098f-40e8-9086-bde5014c0807-kube-api-access-vdpxp\") pod \"aws-ebs-csi-driver-node-zvsxm\" (UID: \"a6f459fa-098f-40e8-9086-bde5014c0807\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zvsxm"
Jul 26 15:50:09.350231 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.349809    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0c48fa94-94f6-43da-af02-7d63be16a9b8-agent-certs\") pod \"konnectivity-agent-pj48f\" (UID: \"0c48fa94-94f6-43da-af02-7d63be16a9b8\") " pod="kube-system/konnectivity-agent-pj48f"
Jul 26 15:50:09.350912 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.349901    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e1f2fc7a-9737-422c-9c17-c8821dc48396-host\") pod \"node-ca-p8kcf\" (UID: \"e1f2fc7a-9737-422c-9c17-c8821dc48396\") " pod="openshift-image-registry/node-ca-p8kcf"
Jul 26 15:50:09.350912 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.349942    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a415587b-d9d6-4f2d-a37b-a49bb34aba81-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-plqcw\" (UID: \"a415587b-d9d6-4f2d-a37b-a49bb34aba81\") " pod="openshift-multus/multus-additional-cni-plugins-plqcw"
Jul 26 15:50:09.350912 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.349955    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e1f2fc7a-9737-422c-9c17-c8821dc48396-host\") pod \"node-ca-p8kcf\" (UID: \"e1f2fc7a-9737-422c-9c17-c8821dc48396\") " pod="openshift-image-registry/node-ca-p8kcf"
Jul 26 15:50:09.350912 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.349963    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/31754f05-c0cf-4fa6-a3b5-f1afff0b7c99-run-systemd\") pod \"ovnkube-node-kwjfd\" (UID: \"31754f05-c0cf-4fa6-a3b5-f1afff0b7c99\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwjfd"
Jul 26 15:50:09.350912 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.349991    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/31754f05-c0cf-4fa6-a3b5-f1afff0b7c99-ovn-node-metrics-cert\") pod \"ovnkube-node-kwjfd\" (UID: \"31754f05-c0cf-4fa6-a3b5-f1afff0b7c99\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwjfd"
Jul 26 15:50:09.350912 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.350011    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0c48fa94-94f6-43da-af02-7d63be16a9b8-konnectivity-ca\") pod \"konnectivity-agent-pj48f\" (UID: \"0c48fa94-94f6-43da-af02-7d63be16a9b8\") " pod="kube-system/konnectivity-agent-pj48f"
Jul 26 15:50:09.350912 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.350044    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/705b8a13-4555-4820-8e91-e2ee6dbac16e-os-release\") pod \"multus-t7zjp\" (UID: \"705b8a13-4555-4820-8e91-e2ee6dbac16e\") " pod="openshift-multus/multus-t7zjp"
Jul 26 15:50:09.350912 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.350077    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/705b8a13-4555-4820-8e91-e2ee6dbac16e-host-var-lib-cni-multus\") pod \"multus-t7zjp\" (UID: \"705b8a13-4555-4820-8e91-e2ee6dbac16e\") " pod="openshift-multus/multus-t7zjp"
Jul 26 15:50:09.350912 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.350114    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5lc8\" (UniqueName: \"kubernetes.io/projected/a415587b-d9d6-4f2d-a37b-a49bb34aba81-kube-api-access-t5lc8\") pod \"multus-additional-cni-plugins-plqcw\" (UID: \"a415587b-d9d6-4f2d-a37b-a49bb34aba81\") " pod="openshift-multus/multus-additional-cni-plugins-plqcw"
Jul 26 15:50:09.350912 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.350158    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/705b8a13-4555-4820-8e91-e2ee6dbac16e-multus-cni-dir\") pod \"multus-t7zjp\" (UID: \"705b8a13-4555-4820-8e91-e2ee6dbac16e\") " pod="openshift-multus/multus-t7zjp"
Jul 26 15:50:09.350912 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.350157    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a415587b-d9d6-4f2d-a37b-a49bb34aba81-cni-binary-copy\") pod \"multus-additional-cni-plugins-plqcw\" (UID: \"a415587b-d9d6-4f2d-a37b-a49bb34aba81\") " pod="openshift-multus/multus-additional-cni-plugins-plqcw"
Jul 26 15:50:09.350912 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.350210    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/705b8a13-4555-4820-8e91-e2ee6dbac16e-multus-socket-dir-parent\") pod \"multus-t7zjp\" (UID: \"705b8a13-4555-4820-8e91-e2ee6dbac16e\") " pod="openshift-multus/multus-t7zjp"
Jul 26 15:50:09.350912 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.350235    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/705b8a13-4555-4820-8e91-e2ee6dbac16e-hostroot\") pod \"multus-t7zjp\" (UID: \"705b8a13-4555-4820-8e91-e2ee6dbac16e\") " pod="openshift-multus/multus-t7zjp"
Jul 26 15:50:09.350912 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.350252    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/705b8a13-4555-4820-8e91-e2ee6dbac16e-etc-kubernetes\") pod \"multus-t7zjp\" (UID: \"705b8a13-4555-4820-8e91-e2ee6dbac16e\") " pod="openshift-multus/multus-t7zjp"
Jul 26 15:50:09.350912 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.350298    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/31754f05-c0cf-4fa6-a3b5-f1afff0b7c99-systemd-units\") pod \"ovnkube-node-kwjfd\" (UID: \"31754f05-c0cf-4fa6-a3b5-f1afff0b7c99\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwjfd"
Jul 26 15:50:09.350912 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.350325    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/31754f05-c0cf-4fa6-a3b5-f1afff0b7c99-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kwjfd\" (UID: \"31754f05-c0cf-4fa6-a3b5-f1afff0b7c99\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwjfd"
Jul 26 15:50:09.350912 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.350347    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e62cc557-6267-4993-84a1-a888ced92345-kubelet-config\") pod \"global-pull-secret-syncer-cxfp4\" (UID: \"e62cc557-6267-4993-84a1-a888ced92345\") " pod="kube-system/global-pull-secret-syncer-cxfp4"
Jul 26 15:50:09.351393 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.350378    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/705b8a13-4555-4820-8e91-e2ee6dbac16e-host-run-netns\") pod \"multus-t7zjp\" (UID: \"705b8a13-4555-4820-8e91-e2ee6dbac16e\") " pod="openshift-multus/multus-t7zjp"
Jul 26 15:50:09.351393 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.350396    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8mlb\" (UniqueName: \"kubernetes.io/projected/705b8a13-4555-4820-8e91-e2ee6dbac16e-kube-api-access-j8mlb\") pod \"multus-t7zjp\" (UID: \"705b8a13-4555-4820-8e91-e2ee6dbac16e\") " pod="openshift-multus/multus-t7zjp"
Jul 26 15:50:09.351393 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.350429    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmxk6\" (UniqueName: \"kubernetes.io/projected/0c90caab-1908-4071-a259-e0343960cbe2-kube-api-access-bmxk6\") pod \"network-check-target-7g75t\" (UID: \"0c90caab-1908-4071-a259-e0343960cbe2\") " pod="openshift-network-diagnostics/network-check-target-7g75t"
Jul 26 15:50:09.351393 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.350450    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/705b8a13-4555-4820-8e91-e2ee6dbac16e-multus-conf-dir\") pod \"multus-t7zjp\" (UID: \"705b8a13-4555-4820-8e91-e2ee6dbac16e\") " pod="openshift-multus/multus-t7zjp"
Jul 26 15:50:09.351393 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.350502    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e1f2fc7a-9737-422c-9c17-c8821dc48396-serviceca\") pod \"node-ca-p8kcf\" (UID: \"e1f2fc7a-9737-422c-9c17-c8821dc48396\") " pod="openshift-image-registry/node-ca-p8kcf"
Jul 26 15:50:09.351393 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.350844    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e1f2fc7a-9737-422c-9c17-c8821dc48396-serviceca\") pod \"node-ca-p8kcf\" (UID: \"e1f2fc7a-9737-422c-9c17-c8821dc48396\") " pod="openshift-image-registry/node-ca-p8kcf"
Jul 26 15:50:09.354077 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.354060    2563 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 26 15:50:09.357372 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.357353    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ms68\" (UniqueName: \"kubernetes.io/projected/e1f2fc7a-9737-422c-9c17-c8821dc48396-kube-api-access-8ms68\") pod \"node-ca-p8kcf\" (UID: \"e1f2fc7a-9737-422c-9c17-c8821dc48396\") " pod="openshift-image-registry/node-ca-p8kcf"
Jul 26 15:50:09.358072 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.358055    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5lc8\" (UniqueName: \"kubernetes.io/projected/a415587b-d9d6-4f2d-a37b-a49bb34aba81-kube-api-access-t5lc8\") pod \"multus-additional-cni-plugins-plqcw\" (UID: \"a415587b-d9d6-4f2d-a37b-a49bb34aba81\") " pod="openshift-multus/multus-additional-cni-plugins-plqcw"
Jul 26 15:50:09.406685 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:09.406655    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6db9e0d726c4428a7173af2b853a8bd3.slice/crio-031037d58cd9131311bbb8fb55a34c729ea77009b0ee2dbb01e5f45f57fc65f8 WatchSource:0}: Error finding container 031037d58cd9131311bbb8fb55a34c729ea77009b0ee2dbb01e5f45f57fc65f8: Status 404 returned error can't find the container with id 031037d58cd9131311bbb8fb55a34c729ea77009b0ee2dbb01e5f45f57fc65f8
Jul 26 15:50:09.410145 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:09.410130    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e83b5d2069ff1f9446be40c52eae0c3.slice/crio-86a6e4c1103c8322898f061ac7666ec7503408f4204e2876227a29129bda22b1 WatchSource:0}: Error finding container 86a6e4c1103c8322898f061ac7666ec7503408f4204e2876227a29129bda22b1: Status 404 returned error can't find the container with id 86a6e4c1103c8322898f061ac7666ec7503408f4204e2876227a29129bda22b1
Jul 26 15:50:09.413763 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.413744    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 26 15:50:09.451822 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.451795    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/31754f05-c0cf-4fa6-a3b5-f1afff0b7c99-var-lib-openvswitch\") pod \"ovnkube-node-kwjfd\" (UID: \"31754f05-c0cf-4fa6-a3b5-f1afff0b7c99\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwjfd"
Jul 26 15:50:09.451822 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.451825    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a59a089c-67e4-455b-9936-af62b889eda9-run\") pod \"tuned-pwcmg\" (UID: \"a59a089c-67e4-455b-9936-af62b889eda9\") " pod="openshift-cluster-node-tuning-operator/tuned-pwcmg"
Jul 26 15:50:09.452009 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.451841    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a59a089c-67e4-455b-9936-af62b889eda9-lib-modules\") pod \"tuned-pwcmg\" (UID: \"a59a089c-67e4-455b-9936-af62b889eda9\") " pod="openshift-cluster-node-tuning-operator/tuned-pwcmg"
Jul 26 15:50:09.452009 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.451856    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a59a089c-67e4-455b-9936-af62b889eda9-etc-tuned\") pod \"tuned-pwcmg\" (UID: \"a59a089c-67e4-455b-9936-af62b889eda9\") " pod="openshift-cluster-node-tuning-operator/tuned-pwcmg"
Jul 26 15:50:09.452009 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.451942    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/31754f05-c0cf-4fa6-a3b5-f1afff0b7c99-var-lib-openvswitch\") pod \"ovnkube-node-kwjfd\" (UID: \"31754f05-c0cf-4fa6-a3b5-f1afff0b7c99\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwjfd"
Jul 26 15:50:09.452009 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.451939    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a6f459fa-098f-40e8-9086-bde5014c0807-device-dir\") pod \"aws-ebs-csi-driver-node-zvsxm\" (UID: \"a6f459fa-098f-40e8-9086-bde5014c0807\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zvsxm"
Jul 26 15:50:09.452009 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.451991    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt8pd\" (UniqueName: \"kubernetes.io/projected/236e97e8-5df0-4a36-9513-a260891d2e9e-kube-api-access-qt8pd\") pod \"iptables-alerter-jltnr\" (UID: \"236e97e8-5df0-4a36-9513-a260891d2e9e\") " pod="openshift-network-operator/iptables-alerter-jltnr"
Jul 26 15:50:09.452009 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.451992    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a59a089c-67e4-455b-9936-af62b889eda9-run\") pod \"tuned-pwcmg\" (UID: \"a59a089c-67e4-455b-9936-af62b889eda9\") " pod="openshift-cluster-node-tuning-operator/tuned-pwcmg"
Jul 26 15:50:09.452009 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.452009    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/31754f05-c0cf-4fa6-a3b5-f1afff0b7c99-node-log\") pod \"ovnkube-node-kwjfd\" (UID: \"31754f05-c0cf-4fa6-a3b5-f1afff0b7c99\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwjfd"
Jul 26 15:50:09.452290 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.451992    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a6f459fa-098f-40e8-9086-bde5014c0807-device-dir\") pod \"aws-ebs-csi-driver-node-zvsxm\" (UID: \"a6f459fa-098f-40e8-9086-bde5014c0807\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zvsxm"
Jul 26 15:50:09.452290 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.452027    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e62cc557-6267-4993-84a1-a888ced92345-dbus\") pod \"global-pull-secret-syncer-cxfp4\" (UID: \"e62cc557-6267-4993-84a1-a888ced92345\") " pod="kube-system/global-pull-secret-syncer-cxfp4"
Jul 26 15:50:09.452290 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.452043    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/aba3d036-e332-4b06-8379-7deb395345ba-hosts-file\") pod \"node-resolver-gnkxb\" (UID: \"aba3d036-e332-4b06-8379-7deb395345ba\") " pod="openshift-dns/node-resolver-gnkxb"
Jul 26 15:50:09.452290 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.451994    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a59a089c-67e4-455b-9936-af62b889eda9-lib-modules\") pod \"tuned-pwcmg\" (UID: \"a59a089c-67e4-455b-9936-af62b889eda9\") " pod="openshift-cluster-node-tuning-operator/tuned-pwcmg"
Jul 26 15:50:09.452290 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.452121    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/aba3d036-e332-4b06-8379-7deb395345ba-hosts-file\") pod \"node-resolver-gnkxb\" (UID: \"aba3d036-e332-4b06-8379-7deb395345ba\") " pod="openshift-dns/node-resolver-gnkxb"
Jul 26 15:50:09.452290 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.452132    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/31754f05-c0cf-4fa6-a3b5-f1afff0b7c99-node-log\") pod \"ovnkube-node-kwjfd\" (UID: \"31754f05-c0cf-4fa6-a3b5-f1afff0b7c99\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwjfd"
Jul 26 15:50:09.452290 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.452058    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/705b8a13-4555-4820-8e91-e2ee6dbac16e-host-run-k8s-cni-cncf-io\") pod \"multus-t7zjp\" (UID: \"705b8a13-4555-4820-8e91-e2ee6dbac16e\") " pod="openshift-multus/multus-t7zjp"
Jul 26 15:50:09.452290 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.452170    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e62cc557-6267-4993-84a1-a888ced92345-dbus\") pod \"global-pull-secret-syncer-cxfp4\" (UID: \"e62cc557-6267-4993-84a1-a888ced92345\") " pod="kube-system/global-pull-secret-syncer-cxfp4"
Jul 26 15:50:09.452290 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.452199    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/705b8a13-4555-4820-8e91-e2ee6dbac16e-host-run-k8s-cni-cncf-io\") pod \"multus-t7zjp\" (UID: \"705b8a13-4555-4820-8e91-e2ee6dbac16e\") " pod="openshift-multus/multus-t7zjp"
Jul 26 15:50:09.452290 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.452259    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/236e97e8-5df0-4a36-9513-a260891d2e9e-iptables-alerter-script\") pod \"iptables-alerter-jltnr\" (UID: \"236e97e8-5df0-4a36-9513-a260891d2e9e\") " pod="openshift-network-operator/iptables-alerter-jltnr"
Jul 26 15:50:09.452676 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.452304    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/236e97e8-5df0-4a36-9513-a260891d2e9e-host-slash\") pod \"iptables-alerter-jltnr\" (UID: \"236e97e8-5df0-4a36-9513-a260891d2e9e\") " pod="openshift-network-operator/iptables-alerter-jltnr"
Jul 26 15:50:09.452676 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.452413    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/236e97e8-5df0-4a36-9513-a260891d2e9e-host-slash\") pod \"iptables-alerter-jltnr\" (UID: \"236e97e8-5df0-4a36-9513-a260891d2e9e\") " pod="openshift-network-operator/iptables-alerter-jltnr"
Jul 26 15:50:09.452676 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.452438    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6rm4\" (UniqueName: \"kubernetes.io/projected/31754f05-c0cf-4fa6-a3b5-f1afff0b7c99-kube-api-access-b6rm4\") pod \"ovnkube-node-kwjfd\" (UID: \"31754f05-c0cf-4fa6-a3b5-f1afff0b7c99\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwjfd"
Jul 26 15:50:09.452676 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.452483    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a59a089c-67e4-455b-9936-af62b889eda9-etc-systemd\") pod \"tuned-pwcmg\" (UID: \"a59a089c-67e4-455b-9936-af62b889eda9\") " pod="openshift-cluster-node-tuning-operator/tuned-pwcmg"
Jul 26 15:50:09.452676 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.452508    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a6f459fa-098f-40e8-9086-bde5014c0807-sys-fs\") pod \"aws-ebs-csi-driver-node-zvsxm\" (UID: \"a6f459fa-098f-40e8-9086-bde5014c0807\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zvsxm"
Jul 26 15:50:09.452676 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.452523    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aba3d036-e332-4b06-8379-7deb395345ba-tmp-dir\") pod \"node-resolver-gnkxb\" (UID: \"aba3d036-e332-4b06-8379-7deb395345ba\") " pod="openshift-dns/node-resolver-gnkxb"
Jul 26 15:50:09.452676 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.452538    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/705b8a13-4555-4820-8e91-e2ee6dbac16e-system-cni-dir\") pod \"multus-t7zjp\" (UID: \"705b8a13-4555-4820-8e91-e2ee6dbac16e\") " pod="openshift-multus/multus-t7zjp"
Jul 26 15:50:09.452676 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.452554    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/31754f05-c0cf-4fa6-a3b5-f1afff0b7c99-run-openvswitch\") pod \"ovnkube-node-kwjfd\" (UID: \"31754f05-c0cf-4fa6-a3b5-f1afff0b7c99\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwjfd"
Jul 26 15:50:09.452676 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.452592    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a6f459fa-098f-40e8-9086-bde5014c0807-sys-fs\") pod \"aws-ebs-csi-driver-node-zvsxm\" (UID: \"a6f459fa-098f-40e8-9086-bde5014c0807\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zvsxm"
Jul 26 15:50:09.452676 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.452654    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a59a089c-67e4-455b-9936-af62b889eda9-etc-sysctl-conf\") pod \"tuned-pwcmg\" (UID: \"a59a089c-67e4-455b-9936-af62b889eda9\") " pod="openshift-cluster-node-tuning-operator/tuned-pwcmg"
Jul 26 15:50:09.452676 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.452679    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/705b8a13-4555-4820-8e91-e2ee6dbac16e-host-var-lib-cni-bin\") pod \"multus-t7zjp\" (UID: \"705b8a13-4555-4820-8e91-e2ee6dbac16e\") " pod="openshift-multus/multus-t7zjp"
Jul 26 15:50:09.453202 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.452690    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/31754f05-c0cf-4fa6-a3b5-f1afff0b7c99-run-openvswitch\") pod \"ovnkube-node-kwjfd\" (UID: \"31754f05-c0cf-4fa6-a3b5-f1afff0b7c99\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwjfd"
Jul 26 15:50:09.453202 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.452709    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca110f78-ba25-4be3-9bed-60e5ef15e2c5-metrics-certs\") pod \"network-metrics-daemon-8gzcl\" (UID: \"ca110f78-ba25-4be3-9bed-60e5ef15e2c5\") " pod="openshift-multus/network-metrics-daemon-8gzcl"
Jul 26 15:50:09.453202 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.452729    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/705b8a13-4555-4820-8e91-e2ee6dbac16e-system-cni-dir\") pod \"multus-t7zjp\" (UID: \"705b8a13-4555-4820-8e91-e2ee6dbac16e\") " pod="openshift-multus/multus-t7zjp"
Jul 26 15:50:09.453202 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.452733    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a59a089c-67e4-455b-9936-af62b889eda9-tmp\") pod \"tuned-pwcmg\" (UID: \"a59a089c-67e4-455b-9936-af62b889eda9\") " pod="openshift-cluster-node-tuning-operator/tuned-pwcmg"
Jul 26 15:50:09.453202 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.452730    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/705b8a13-4555-4820-8e91-e2ee6dbac16e-host-var-lib-cni-bin\") pod \"multus-t7zjp\" (UID: \"705b8a13-4555-4820-8e91-e2ee6dbac16e\") " pod="openshift-multus/multus-t7zjp"
Jul 26 15:50:09.453202 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.452759    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a6f459fa-098f-40e8-9086-bde5014c0807-registration-dir\") pod \"aws-ebs-csi-driver-node-zvsxm\" (UID: \"a6f459fa-098f-40e8-9086-bde5014c0807\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zvsxm"
Jul 26 15:50:09.453202 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.452785    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/705b8a13-4555-4820-8e91-e2ee6dbac16e-multus-daemon-config\") pod \"multus-t7zjp\" (UID: \"705b8a13-4555-4820-8e91-e2ee6dbac16e\") " pod="openshift-multus/multus-t7zjp"
Jul 26 15:50:09.453202 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.452828    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/31754f05-c0cf-4fa6-a3b5-f1afff0b7c99-host-slash\") pod \"ovnkube-node-kwjfd\" (UID: \"31754f05-c0cf-4fa6-a3b5-f1afff0b7c99\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwjfd"
Jul 26 15:50:09.453202 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:09.452834    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 26 15:50:09.453202 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.452822    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a59a089c-67e4-455b-9936-af62b889eda9-etc-systemd\") pod \"tuned-pwcmg\" (UID: \"a59a089c-67e4-455b-9936-af62b889eda9\") " pod="openshift-cluster-node-tuning-operator/tuned-pwcmg"
Jul 26 15:50:09.453202 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.452854    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a59a089c-67e4-455b-9936-af62b889eda9-etc-sysctl-d\") pod \"tuned-pwcmg\" (UID: \"a59a089c-67e4-455b-9936-af62b889eda9\") " pod="openshift-cluster-node-tuning-operator/tuned-pwcmg"
Jul 26 15:50:09.453202 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.452878    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8zd9\" (UniqueName: \"kubernetes.io/projected/a59a089c-67e4-455b-9936-af62b889eda9-kube-api-access-k8zd9\") pod \"tuned-pwcmg\" (UID: \"a59a089c-67e4-455b-9936-af62b889eda9\") " pod="openshift-cluster-node-tuning-operator/tuned-pwcmg"
Jul 26 15:50:09.453202 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.452901    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/236e97e8-5df0-4a36-9513-a260891d2e9e-iptables-alerter-script\") pod \"iptables-alerter-jltnr\" (UID: \"236e97e8-5df0-4a36-9513-a260891d2e9e\") " pod="openshift-network-operator/iptables-alerter-jltnr"
Jul 26 15:50:09.453202 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:09.452919    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca110f78-ba25-4be3-9bed-60e5ef15e2c5-metrics-certs podName:ca110f78-ba25-4be3-9bed-60e5ef15e2c5 nodeName:}" failed. No retries permitted until 2026-07-26 15:50:09.952880511 +0000 UTC m=+2.057106323 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca110f78-ba25-4be3-9bed-60e5ef15e2c5-metrics-certs") pod "network-metrics-daemon-8gzcl" (UID: "ca110f78-ba25-4be3-9bed-60e5ef15e2c5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 26 15:50:09.453202 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.452941    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aba3d036-e332-4b06-8379-7deb395345ba-tmp-dir\") pod \"node-resolver-gnkxb\" (UID: \"aba3d036-e332-4b06-8379-7deb395345ba\") " pod="openshift-dns/node-resolver-gnkxb"
Jul 26 15:50:09.453202 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.452959    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwnhx\" (UniqueName: \"kubernetes.io/projected/aba3d036-e332-4b06-8379-7deb395345ba-kube-api-access-wwnhx\") pod \"node-resolver-gnkxb\" (UID: \"aba3d036-e332-4b06-8379-7deb395345ba\") " pod="openshift-dns/node-resolver-gnkxb"
Jul 26 15:50:09.453202 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.452975    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a59a089c-67e4-455b-9936-af62b889eda9-etc-sysctl-conf\") pod \"tuned-pwcmg\" (UID: \"a59a089c-67e4-455b-9936-af62b889eda9\") " pod="openshift-cluster-node-tuning-operator/tuned-pwcmg"
Jul 26 15:50:09.454112 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.452994    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a59a089c-67e4-455b-9936-af62b889eda9-etc-modprobe-d\") pod \"tuned-pwcmg\" (UID: \"a59a089c-67e4-455b-9936-af62b889eda9\") " pod="openshift-cluster-node-tuning-operator/tuned-pwcmg"
Jul 26 15:50:09.454112 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.453047    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a6f459fa-098f-40e8-9086-bde5014c0807-registration-dir\") pod \"aws-ebs-csi-driver-node-zvsxm\" (UID: \"a6f459fa-098f-40e8-9086-bde5014c0807\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zvsxm"
Jul 26 15:50:09.454112 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.453076    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a59a089c-67e4-455b-9936-af62b889eda9-etc-sysconfig\") pod \"tuned-pwcmg\" (UID: \"a59a089c-67e4-455b-9936-af62b889eda9\") " pod="openshift-cluster-node-tuning-operator/tuned-pwcmg"
Jul 26 15:50:09.454112 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.453102    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a59a089c-67e4-455b-9936-af62b889eda9-sys\") pod \"tuned-pwcmg\" (UID: \"a59a089c-67e4-455b-9936-af62b889eda9\") " pod="openshift-cluster-node-tuning-operator/tuned-pwcmg"
Jul 26 15:50:09.454112 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.453105    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a59a089c-67e4-455b-9936-af62b889eda9-etc-sysctl-d\") pod \"tuned-pwcmg\" (UID: \"a59a089c-67e4-455b-9936-af62b889eda9\") " pod="openshift-cluster-node-tuning-operator/tuned-pwcmg"
Jul 26 15:50:09.454112 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.453125    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a59a089c-67e4-455b-9936-af62b889eda9-host\") pod \"tuned-pwcmg\" (UID: \"a59a089c-67e4-455b-9936-af62b889eda9\") " pod="openshift-cluster-node-tuning-operator/tuned-pwcmg"
Jul 26 15:50:09.454112 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.453138    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/31754f05-c0cf-4fa6-a3b5-f1afff0b7c99-host-slash\") pod \"ovnkube-node-kwjfd\" (UID: \"31754f05-c0cf-4fa6-a3b5-f1afff0b7c99\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwjfd"
Jul 26 15:50:09.454112 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.453146    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a59a089c-67e4-455b-9936-af62b889eda9-etc-sysconfig\") pod \"tuned-pwcmg\" (UID: \"a59a089c-67e4-455b-9936-af62b889eda9\") " pod="openshift-cluster-node-tuning-operator/tuned-pwcmg"
Jul 26 15:50:09.454112 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.453151    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/705b8a13-4555-4820-8e91-e2ee6dbac16e-cni-binary-copy\") pod \"multus-t7zjp\" (UID: \"705b8a13-4555-4820-8e91-e2ee6dbac16e\") " pod="openshift-multus/multus-t7zjp"
Jul 26 15:50:09.454112 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.453160    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a59a089c-67e4-455b-9936-af62b889eda9-sys\") pod \"tuned-pwcmg\" (UID: \"a59a089c-67e4-455b-9936-af62b889eda9\") " pod="openshift-cluster-node-tuning-operator/tuned-pwcmg"
Jul 26 15:50:09.454112 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.453182    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/31754f05-c0cf-4fa6-a3b5-f1afff0b7c99-etc-openvswitch\") pod \"ovnkube-node-kwjfd\" (UID: \"31754f05-c0cf-4fa6-a3b5-f1afff0b7c99\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwjfd"
Jul 26 15:50:09.454112 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.453196    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a59a089c-67e4-455b-9936-af62b889eda9-host\") pod \"tuned-pwcmg\" (UID: \"a59a089c-67e4-455b-9936-af62b889eda9\") " pod="openshift-cluster-node-tuning-operator/tuned-pwcmg"
Jul 26 15:50:09.454112 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.453206    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/31754f05-c0cf-4fa6-a3b5-f1afff0b7c99-run-ovn\") pod \"ovnkube-node-kwjfd\" (UID: \"31754f05-c0cf-4fa6-a3b5-f1afff0b7c99\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwjfd"
Jul 26 15:50:09.454112 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.453243    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/31754f05-c0cf-4fa6-a3b5-f1afff0b7c99-run-ovn\") pod \"ovnkube-node-kwjfd\" (UID: \"31754f05-c0cf-4fa6-a3b5-f1afff0b7c99\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwjfd"
Jul 26 15:50:09.454112 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.453249    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/31754f05-c0cf-4fa6-a3b5-f1afff0b7c99-etc-openvswitch\") pod \"ovnkube-node-kwjfd\" (UID: \"31754f05-c0cf-4fa6-a3b5-f1afff0b7c99\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwjfd"
Jul 26 15:50:09.454112 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.453280    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/31754f05-c0cf-4fa6-a3b5-f1afff0b7c99-host-cni-bin\") pod \"ovnkube-node-kwjfd\" (UID: \"31754f05-c0cf-4fa6-a3b5-f1afff0b7c99\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwjfd"
Jul 26 15:50:09.454112 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.453288    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a59a089c-67e4-455b-9936-af62b889eda9-etc-modprobe-d\") pod \"tuned-pwcmg\" (UID: \"a59a089c-67e4-455b-9936-af62b889eda9\") " pod="openshift-cluster-node-tuning-operator/tuned-pwcmg"
Jul 26 15:50:09.454112 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.453317    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/31754f05-c0cf-4fa6-a3b5-f1afff0b7c99-host-cni-netd\") pod \"ovnkube-node-kwjfd\" (UID: \"31754f05-c0cf-4fa6-a3b5-f1afff0b7c99\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwjfd"
Jul 26 15:50:09.454987 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.453343    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a59a089c-67e4-455b-9936-af62b889eda9-etc-kubernetes\") pod \"tuned-pwcmg\" (UID: \"a59a089c-67e4-455b-9936-af62b889eda9\") " pod="openshift-cluster-node-tuning-operator/tuned-pwcmg"
Jul 26 15:50:09.454987 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.453352    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/31754f05-c0cf-4fa6-a3b5-f1afff0b7c99-host-cni-bin\") pod \"ovnkube-node-kwjfd\" (UID: \"31754f05-c0cf-4fa6-a3b5-f1afff0b7c99\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwjfd"
Jul 26 15:50:09.454987 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.453367    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a59a089c-67e4-455b-9936-af62b889eda9-var-lib-kubelet\") pod \"tuned-pwcmg\" (UID: \"a59a089c-67e4-455b-9936-af62b889eda9\") " pod="openshift-cluster-node-tuning-operator/tuned-pwcmg"
Jul 26 15:50:09.454987 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.453373    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/31754f05-c0cf-4fa6-a3b5-f1afff0b7c99-host-cni-netd\") pod \"ovnkube-node-kwjfd\" (UID: \"31754f05-c0cf-4fa6-a3b5-f1afff0b7c99\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwjfd"
Jul 26 15:50:09.454987 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.453398    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a59a089c-67e4-455b-9936-af62b889eda9-etc-kubernetes\") pod \"tuned-pwcmg\" (UID: \"a59a089c-67e4-455b-9936-af62b889eda9\") " pod="openshift-cluster-node-tuning-operator/tuned-pwcmg"
Jul 26 15:50:09.454987 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.453398    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/31754f05-c0cf-4fa6-a3b5-f1afff0b7c99-log-socket\") pod \"ovnkube-node-kwjfd\" (UID: \"31754f05-c0cf-4fa6-a3b5-f1afff0b7c99\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwjfd"
Jul 26 15:50:09.454987 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.453435    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a59a089c-67e4-455b-9936-af62b889eda9-var-lib-kubelet\") pod \"tuned-pwcmg\" (UID: \"a59a089c-67e4-455b-9936-af62b889eda9\") " pod="openshift-cluster-node-tuning-operator/tuned-pwcmg"
Jul 26 15:50:09.454987 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.453451    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/31754f05-c0cf-4fa6-a3b5-f1afff0b7c99-host-run-ovn-kubernetes\") pod \"ovnkube-node-kwjfd\" (UID: \"31754f05-c0cf-4fa6-a3b5-f1afff0b7c99\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwjfd"
Jul 26 15:50:09.454987 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.453464    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/31754f05-c0cf-4fa6-a3b5-f1afff0b7c99-log-socket\") pod \"ovnkube-node-kwjfd\" (UID: \"31754f05-c0cf-4fa6-a3b5-f1afff0b7c99\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwjfd"
Jul 26 15:50:09.454987 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.453492    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/31754f05-c0cf-4fa6-a3b5-f1afff0b7c99-ovnkube-script-lib\") pod \"ovnkube-node-kwjfd\" (UID: \"31754f05-c0cf-4fa6-a3b5-f1afff0b7c99\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwjfd"
Jul 26 15:50:09.454987 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.453521    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdsd2\" (UniqueName: \"kubernetes.io/projected/ca110f78-ba25-4be3-9bed-60e5ef15e2c5-kube-api-access-jdsd2\") pod \"network-metrics-daemon-8gzcl\" (UID: \"ca110f78-ba25-4be3-9bed-60e5ef15e2c5\") " pod="openshift-multus/network-metrics-daemon-8gzcl"
Jul 26 15:50:09.454987 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.453547    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a6f459fa-098f-40e8-9086-bde5014c0807-etc-selinux\") pod \"aws-ebs-csi-driver-node-zvsxm\" (UID: \"a6f459fa-098f-40e8-9086-bde5014c0807\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zvsxm"
Jul 26 15:50:09.454987 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.453573    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdpxp\" (UniqueName: \"kubernetes.io/projected/a6f459fa-098f-40e8-9086-bde5014c0807-kube-api-access-vdpxp\") pod \"aws-ebs-csi-driver-node-zvsxm\" (UID: \"a6f459fa-098f-40e8-9086-bde5014c0807\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zvsxm"
Jul 26 15:50:09.454987 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.453597    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0c48fa94-94f6-43da-af02-7d63be16a9b8-agent-certs\") pod \"konnectivity-agent-pj48f\" (UID: \"0c48fa94-94f6-43da-af02-7d63be16a9b8\") " pod="kube-system/konnectivity-agent-pj48f"
Jul 26 15:50:09.454987 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.453625    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/31754f05-c0cf-4fa6-a3b5-f1afff0b7c99-run-systemd\") pod \"ovnkube-node-kwjfd\" (UID: \"31754f05-c0cf-4fa6-a3b5-f1afff0b7c99\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwjfd"
Jul 26 15:50:09.454987 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.453649    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/31754f05-c0cf-4fa6-a3b5-f1afff0b7c99-ovn-node-metrics-cert\") pod \"ovnkube-node-kwjfd\" (UID: \"31754f05-c0cf-4fa6-a3b5-f1afff0b7c99\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwjfd"
Jul 26 15:50:09.454987 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.453676    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0c48fa94-94f6-43da-af02-7d63be16a9b8-konnectivity-ca\") pod \"konnectivity-agent-pj48f\" (UID: \"0c48fa94-94f6-43da-af02-7d63be16a9b8\") " pod="kube-system/konnectivity-agent-pj48f"
Jul 26 15:50:09.455574 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.453701    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/705b8a13-4555-4820-8e91-e2ee6dbac16e-os-release\") pod \"multus-t7zjp\" (UID: \"705b8a13-4555-4820-8e91-e2ee6dbac16e\") " pod="openshift-multus/multus-t7zjp"
Jul 26 15:50:09.455574 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.453727    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/705b8a13-4555-4820-8e91-e2ee6dbac16e-host-var-lib-cni-multus\") pod \"multus-t7zjp\" (UID: \"705b8a13-4555-4820-8e91-e2ee6dbac16e\") " pod="openshift-multus/multus-t7zjp"
Jul 26 15:50:09.455574 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.453756    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/705b8a13-4555-4820-8e91-e2ee6dbac16e-multus-cni-dir\") pod \"multus-t7zjp\" (UID: \"705b8a13-4555-4820-8e91-e2ee6dbac16e\") " pod="openshift-multus/multus-t7zjp"
Jul 26 15:50:09.455574 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.453745    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/705b8a13-4555-4820-8e91-e2ee6dbac16e-cni-binary-copy\") pod \"multus-t7zjp\" (UID: \"705b8a13-4555-4820-8e91-e2ee6dbac16e\") " pod="openshift-multus/multus-t7zjp"
Jul 26 15:50:09.455574 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.453781    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/705b8a13-4555-4820-8e91-e2ee6dbac16e-multus-socket-dir-parent\") pod \"multus-t7zjp\" (UID: \"705b8a13-4555-4820-8e91-e2ee6dbac16e\") " pod="openshift-multus/multus-t7zjp"
Jul 26 15:50:09.455574 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.453801    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/705b8a13-4555-4820-8e91-e2ee6dbac16e-multus-daemon-config\") pod \"multus-t7zjp\" (UID: \"705b8a13-4555-4820-8e91-e2ee6dbac16e\") " pod="openshift-multus/multus-t7zjp"
Jul 26 15:50:09.455574 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.453874    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/31754f05-c0cf-4fa6-a3b5-f1afff0b7c99-run-systemd\") pod \"ovnkube-node-kwjfd\" (UID: \"31754f05-c0cf-4fa6-a3b5-f1afff0b7c99\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwjfd"
Jul 26 15:50:09.455574 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.453893    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a6f459fa-098f-40e8-9086-bde5014c0807-etc-selinux\") pod \"aws-ebs-csi-driver-node-zvsxm\" (UID: \"a6f459fa-098f-40e8-9086-bde5014c0807\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zvsxm"
Jul 26 15:50:09.455574 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.453926    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/705b8a13-4555-4820-8e91-e2ee6dbac16e-hostroot\") pod \"multus-t7zjp\" (UID: \"705b8a13-4555-4820-8e91-e2ee6dbac16e\") " pod="openshift-multus/multus-t7zjp"
Jul 26 15:50:09.455574 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.453936    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/705b8a13-4555-4820-8e91-e2ee6dbac16e-multus-socket-dir-parent\") pod \"multus-t7zjp\" (UID: \"705b8a13-4555-4820-8e91-e2ee6dbac16e\") " pod="openshift-multus/multus-t7zjp"
Jul 26 15:50:09.455574 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.453955    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/705b8a13-4555-4820-8e91-e2ee6dbac16e-etc-kubernetes\") pod \"multus-t7zjp\" (UID: \"705b8a13-4555-4820-8e91-e2ee6dbac16e\") " pod="openshift-multus/multus-t7zjp"
Jul 26 15:50:09.455574 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.453993    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/31754f05-c0cf-4fa6-a3b5-f1afff0b7c99-systemd-units\") pod \"ovnkube-node-kwjfd\" (UID: \"31754f05-c0cf-4fa6-a3b5-f1afff0b7c99\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwjfd"
Jul 26 15:50:09.455574 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.454017    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/31754f05-c0cf-4fa6-a3b5-f1afff0b7c99-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kwjfd\" (UID: \"31754f05-c0cf-4fa6-a3b5-f1afff0b7c99\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwjfd"
Jul 26 15:50:09.455574 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.454048    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e62cc557-6267-4993-84a1-a888ced92345-kubelet-config\") pod \"global-pull-secret-syncer-cxfp4\" (UID: \"e62cc557-6267-4993-84a1-a888ced92345\") " pod="kube-system/global-pull-secret-syncer-cxfp4"
Jul 26 15:50:09.455574 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.454052    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/705b8a13-4555-4820-8e91-e2ee6dbac16e-os-release\") pod \"multus-t7zjp\" (UID: \"705b8a13-4555-4820-8e91-e2ee6dbac16e\") " pod="openshift-multus/multus-t7zjp"
Jul 26 15:50:09.455574 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.454081    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/705b8a13-4555-4820-8e91-e2ee6dbac16e-host-run-netns\") pod \"multus-t7zjp\" (UID: \"705b8a13-4555-4820-8e91-e2ee6dbac16e\") " pod="openshift-multus/multus-t7zjp"
Jul 26 15:50:09.455574 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.454113    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8mlb\" (UniqueName: \"kubernetes.io/projected/705b8a13-4555-4820-8e91-e2ee6dbac16e-kube-api-access-j8mlb\") pod \"multus-t7zjp\" (UID: \"705b8a13-4555-4820-8e91-e2ee6dbac16e\") " pod="openshift-multus/multus-t7zjp"
Jul 26 15:50:09.455574 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.454124    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/705b8a13-4555-4820-8e91-e2ee6dbac16e-multus-cni-dir\") pod \"multus-t7zjp\" (UID: \"705b8a13-4555-4820-8e91-e2ee6dbac16e\") " pod="openshift-multus/multus-t7zjp"
Jul 26 15:50:09.456182 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.454148    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmxk6\" (UniqueName: \"kubernetes.io/projected/0c90caab-1908-4071-a259-e0343960cbe2-kube-api-access-bmxk6\") pod \"network-check-target-7g75t\" (UID: \"0c90caab-1908-4071-a259-e0343960cbe2\") " pod="openshift-network-diagnostics/network-check-target-7g75t"
Jul 26 15:50:09.456182 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.453994    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/705b8a13-4555-4820-8e91-e2ee6dbac16e-host-var-lib-cni-multus\") pod \"multus-t7zjp\" (UID: \"705b8a13-4555-4820-8e91-e2ee6dbac16e\") " pod="openshift-multus/multus-t7zjp"
Jul 26 15:50:09.456182 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.453521    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/31754f05-c0cf-4fa6-a3b5-f1afff0b7c99-host-run-ovn-kubernetes\") pod \"ovnkube-node-kwjfd\" (UID: \"31754f05-c0cf-4fa6-a3b5-f1afff0b7c99\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwjfd"
Jul 26 15:50:09.456182 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.454181    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/705b8a13-4555-4820-8e91-e2ee6dbac16e-multus-conf-dir\") pod \"multus-t7zjp\" (UID: \"705b8a13-4555-4820-8e91-e2ee6dbac16e\") " pod="openshift-multus/multus-t7zjp"
Jul 26 15:50:09.456182 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.454214    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/31754f05-c0cf-4fa6-a3b5-f1afff0b7c99-env-overrides\") pod \"ovnkube-node-kwjfd\" (UID: \"31754f05-c0cf-4fa6-a3b5-f1afff0b7c99\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwjfd"
Jul 26 15:50:09.456182 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.454241    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e62cc557-6267-4993-84a1-a888ced92345-original-pull-secret\") pod \"global-pull-secret-syncer-cxfp4\" (UID: \"e62cc557-6267-4993-84a1-a888ced92345\") " pod="kube-system/global-pull-secret-syncer-cxfp4"
Jul 26 15:50:09.456182 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.454246    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/705b8a13-4555-4820-8e91-e2ee6dbac16e-etc-kubernetes\") pod \"multus-t7zjp\" (UID: \"705b8a13-4555-4820-8e91-e2ee6dbac16e\") " pod="openshift-multus/multus-t7zjp"
Jul 26 15:50:09.456182 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.454264    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a6f459fa-098f-40e8-9086-bde5014c0807-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zvsxm\" (UID: \"a6f459fa-098f-40e8-9086-bde5014c0807\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zvsxm"
Jul 26 15:50:09.456182 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.454288    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/31754f05-c0cf-4fa6-a3b5-f1afff0b7c99-systemd-units\") pod \"ovnkube-node-kwjfd\" (UID: \"31754f05-c0cf-4fa6-a3b5-f1afff0b7c99\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwjfd"
Jul 26 15:50:09.456182 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.454290    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a6f459fa-098f-40e8-9086-bde5014c0807-socket-dir\") pod \"aws-ebs-csi-driver-node-zvsxm\" (UID: \"a6f459fa-098f-40e8-9086-bde5014c0807\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zvsxm"
Jul 26 15:50:09.456182 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.454334    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/705b8a13-4555-4820-8e91-e2ee6dbac16e-cnibin\") pod \"multus-t7zjp\" (UID: \"705b8a13-4555-4820-8e91-e2ee6dbac16e\") " pod="openshift-multus/multus-t7zjp"
Jul 26 15:50:09.456182 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.454373    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/705b8a13-4555-4820-8e91-e2ee6dbac16e-host-var-lib-kubelet\") pod \"multus-t7zjp\" (UID: \"705b8a13-4555-4820-8e91-e2ee6dbac16e\") " pod="openshift-multus/multus-t7zjp"
Jul 26 15:50:09.456182 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.454396    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a6f459fa-098f-40e8-9086-bde5014c0807-socket-dir\") pod \"aws-ebs-csi-driver-node-zvsxm\" (UID: \"a6f459fa-098f-40e8-9086-bde5014c0807\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zvsxm"
Jul 26 15:50:09.456182 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.454403    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/31754f05-c0cf-4fa6-a3b5-f1afff0b7c99-host-kubelet\") pod \"ovnkube-node-kwjfd\" (UID: \"31754f05-c0cf-4fa6-a3b5-f1afff0b7c99\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwjfd"
Jul 26 15:50:09.456182 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.454454    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/31754f05-c0cf-4fa6-a3b5-f1afff0b7c99-host-run-netns\") pod \"ovnkube-node-kwjfd\" (UID: \"31754f05-c0cf-4fa6-a3b5-f1afff0b7c99\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwjfd"
Jul 26 15:50:09.456182 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.454453    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/705b8a13-4555-4820-8e91-e2ee6dbac16e-multus-conf-dir\") pod \"multus-t7zjp\" (UID: \"705b8a13-4555-4820-8e91-e2ee6dbac16e\") " pod="openshift-multus/multus-t7zjp"
Jul 26 15:50:09.456182 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.454528    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/705b8a13-4555-4820-8e91-e2ee6dbac16e-cnibin\") pod \"multus-t7zjp\" (UID: \"705b8a13-4555-4820-8e91-e2ee6dbac16e\") " pod="openshift-multus/multus-t7zjp"
Jul 26 15:50:09.456182 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.454569    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/705b8a13-4555-4820-8e91-e2ee6dbac16e-host-var-lib-kubelet\") pod \"multus-t7zjp\" (UID: \"705b8a13-4555-4820-8e91-e2ee6dbac16e\") " pod="openshift-multus/multus-t7zjp"
Jul 26 15:50:09.456714 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.454609    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/31754f05-c0cf-4fa6-a3b5-f1afff0b7c99-host-kubelet\") pod \"ovnkube-node-kwjfd\" (UID: \"31754f05-c0cf-4fa6-a3b5-f1afff0b7c99\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwjfd"
Jul 26 15:50:09.456714 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.454215    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/705b8a13-4555-4820-8e91-e2ee6dbac16e-hostroot\") pod \"multus-t7zjp\" (UID: \"705b8a13-4555-4820-8e91-e2ee6dbac16e\") " pod="openshift-multus/multus-t7zjp"
Jul 26 15:50:09.456714 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.454626    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0c48fa94-94f6-43da-af02-7d63be16a9b8-konnectivity-ca\") pod \"konnectivity-agent-pj48f\" (UID: \"0c48fa94-94f6-43da-af02-7d63be16a9b8\") " pod="kube-system/konnectivity-agent-pj48f"
Jul 26 15:50:09.456714 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.454636    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/31754f05-c0cf-4fa6-a3b5-f1afff0b7c99-host-run-netns\") pod \"ovnkube-node-kwjfd\" (UID: \"31754f05-c0cf-4fa6-a3b5-f1afff0b7c99\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwjfd"
Jul 26 15:50:09.456714 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.454674    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e62cc557-6267-4993-84a1-a888ced92345-kubelet-config\") pod \"global-pull-secret-syncer-cxfp4\" (UID: \"e62cc557-6267-4993-84a1-a888ced92345\") " pod="kube-system/global-pull-secret-syncer-cxfp4"
Jul 26 15:50:09.456714 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:09.454686    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 26 15:50:09.456714 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.454696    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/705b8a13-4555-4820-8e91-e2ee6dbac16e-host-run-netns\") pod \"multus-t7zjp\" (UID: \"705b8a13-4555-4820-8e91-e2ee6dbac16e\") " pod="openshift-multus/multus-t7zjp"
Jul 26 15:50:09.456714 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.454729    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/31754f05-c0cf-4fa6-a3b5-f1afff0b7c99-ovnkube-config\") pod \"ovnkube-node-kwjfd\" (UID: \"31754f05-c0cf-4fa6-a3b5-f1afff0b7c99\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwjfd"
Jul 26 15:50:09.456714 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.454737    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/31754f05-c0cf-4fa6-a3b5-f1afff0b7c99-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kwjfd\" (UID: \"31754f05-c0cf-4fa6-a3b5-f1afff0b7c99\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwjfd"
Jul 26 15:50:09.456714 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:09.454786    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e62cc557-6267-4993-84a1-a888ced92345-original-pull-secret podName:e62cc557-6267-4993-84a1-a888ced92345 nodeName:}" failed. No retries permitted until 2026-07-26 15:50:09.954731861 +0000 UTC m=+2.058957654 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e62cc557-6267-4993-84a1-a888ced92345-original-pull-secret") pod "global-pull-secret-syncer-cxfp4" (UID: "e62cc557-6267-4993-84a1-a888ced92345") : object "kube-system"/"original-pull-secret" not registered
Jul 26 15:50:09.456714 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.454790    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/31754f05-c0cf-4fa6-a3b5-f1afff0b7c99-ovnkube-script-lib\") pod \"ovnkube-node-kwjfd\" (UID: \"31754f05-c0cf-4fa6-a3b5-f1afff0b7c99\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwjfd"
Jul 26 15:50:09.456714 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.454818    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a6f459fa-098f-40e8-9086-bde5014c0807-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zvsxm\" (UID: \"a6f459fa-098f-40e8-9086-bde5014c0807\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zvsxm"
Jul 26 15:50:09.456714 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.454828    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/705b8a13-4555-4820-8e91-e2ee6dbac16e-host-run-multus-certs\") pod \"multus-t7zjp\" (UID: \"705b8a13-4555-4820-8e91-e2ee6dbac16e\") " pod="openshift-multus/multus-t7zjp"
Jul 26 15:50:09.456714 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.455006    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/705b8a13-4555-4820-8e91-e2ee6dbac16e-host-run-multus-certs\") pod \"multus-t7zjp\" (UID: \"705b8a13-4555-4820-8e91-e2ee6dbac16e\") " pod="openshift-multus/multus-t7zjp"
Jul 26 15:50:09.456714 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.455093    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a59a089c-67e4-455b-9936-af62b889eda9-etc-tuned\") pod \"tuned-pwcmg\" (UID: \"a59a089c-67e4-455b-9936-af62b889eda9\") " pod="openshift-cluster-node-tuning-operator/tuned-pwcmg"
Jul 26 15:50:09.456714 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.455330    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/31754f05-c0cf-4fa6-a3b5-f1afff0b7c99-env-overrides\") pod \"ovnkube-node-kwjfd\" (UID: \"31754f05-c0cf-4fa6-a3b5-f1afff0b7c99\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwjfd"
Jul 26 15:50:09.456714 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.455352    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a59a089c-67e4-455b-9936-af62b889eda9-tmp\") pod \"tuned-pwcmg\" (UID: \"a59a089c-67e4-455b-9936-af62b889eda9\") " pod="openshift-cluster-node-tuning-operator/tuned-pwcmg"
Jul 26 15:50:09.457235 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.455384    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/31754f05-c0cf-4fa6-a3b5-f1afff0b7c99-ovnkube-config\") pod \"ovnkube-node-kwjfd\" (UID: \"31754f05-c0cf-4fa6-a3b5-f1afff0b7c99\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwjfd"
Jul 26 15:50:09.457235 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.456329    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0c48fa94-94f6-43da-af02-7d63be16a9b8-agent-certs\") pod \"konnectivity-agent-pj48f\" (UID: \"0c48fa94-94f6-43da-af02-7d63be16a9b8\") " pod="kube-system/konnectivity-agent-pj48f"
Jul 26 15:50:09.457235 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.456806    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/31754f05-c0cf-4fa6-a3b5-f1afff0b7c99-ovn-node-metrics-cert\") pod \"ovnkube-node-kwjfd\" (UID: \"31754f05-c0cf-4fa6-a3b5-f1afff0b7c99\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwjfd"
Jul 26 15:50:09.464655 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:09.464631    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 26 15:50:09.464655 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:09.464650    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 26 15:50:09.464655 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:09.464660    2563 projected.go:196] Error preparing data for projected volume kube-api-access-bmxk6 for pod openshift-network-diagnostics/network-check-target-7g75t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 26 15:50:09.464901 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:09.464706    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0c90caab-1908-4071-a259-e0343960cbe2-kube-api-access-bmxk6 podName:0c90caab-1908-4071-a259-e0343960cbe2 nodeName:}" failed. No retries permitted until 2026-07-26 15:50:09.964693811 +0000 UTC m=+2.068919601 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-bmxk6" (UniqueName: "kubernetes.io/projected/0c90caab-1908-4071-a259-e0343960cbe2-kube-api-access-bmxk6") pod "network-check-target-7g75t" (UID: "0c90caab-1908-4071-a259-e0343960cbe2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 26 15:50:09.467490 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.467463    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8zd9\" (UniqueName: \"kubernetes.io/projected/a59a089c-67e4-455b-9936-af62b889eda9-kube-api-access-k8zd9\") pod \"tuned-pwcmg\" (UID: \"a59a089c-67e4-455b-9936-af62b889eda9\") " pod="openshift-cluster-node-tuning-operator/tuned-pwcmg"
Jul 26 15:50:09.468105 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.468077    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdsd2\" (UniqueName: \"kubernetes.io/projected/ca110f78-ba25-4be3-9bed-60e5ef15e2c5-kube-api-access-jdsd2\") pod \"network-metrics-daemon-8gzcl\" (UID: \"ca110f78-ba25-4be3-9bed-60e5ef15e2c5\") " pod="openshift-multus/network-metrics-daemon-8gzcl"
Jul 26 15:50:09.468278 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.468256    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt8pd\" (UniqueName: \"kubernetes.io/projected/236e97e8-5df0-4a36-9513-a260891d2e9e-kube-api-access-qt8pd\") pod \"iptables-alerter-jltnr\" (UID: \"236e97e8-5df0-4a36-9513-a260891d2e9e\") " pod="openshift-network-operator/iptables-alerter-jltnr"
Jul 26 15:50:09.468545 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.468524    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdpxp\" (UniqueName: \"kubernetes.io/projected/a6f459fa-098f-40e8-9086-bde5014c0807-kube-api-access-vdpxp\") pod \"aws-ebs-csi-driver-node-zvsxm\" (UID: \"a6f459fa-098f-40e8-9086-bde5014c0807\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zvsxm"
Jul 26 15:50:09.469790 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.469756    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6rm4\" (UniqueName: \"kubernetes.io/projected/31754f05-c0cf-4fa6-a3b5-f1afff0b7c99-kube-api-access-b6rm4\") pod \"ovnkube-node-kwjfd\" (UID: \"31754f05-c0cf-4fa6-a3b5-f1afff0b7c99\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwjfd"
Jul 26 15:50:09.469790 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.469762    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8mlb\" (UniqueName: \"kubernetes.io/projected/705b8a13-4555-4820-8e91-e2ee6dbac16e-kube-api-access-j8mlb\") pod \"multus-t7zjp\" (UID: \"705b8a13-4555-4820-8e91-e2ee6dbac16e\") " pod="openshift-multus/multus-t7zjp"
Jul 26 15:50:09.470696 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.470667    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwnhx\" (UniqueName: \"kubernetes.io/projected/aba3d036-e332-4b06-8379-7deb395345ba-kube-api-access-wwnhx\") pod \"node-resolver-gnkxb\" (UID: \"aba3d036-e332-4b06-8379-7deb395345ba\") " pod="openshift-dns/node-resolver-gnkxb"
Jul 26 15:50:09.560592 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.560514    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-p8kcf"
Jul 26 15:50:09.570754 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:09.570725    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1f2fc7a_9737_422c_9c17_c8821dc48396.slice/crio-b6d476cfa89f133babbe1b2f4f595bd8f4c476597f0a7c5f98e8ab4657ff7dc9 WatchSource:0}: Error finding container b6d476cfa89f133babbe1b2f4f595bd8f4c476597f0a7c5f98e8ab4657ff7dc9: Status 404 returned error can't find the container with id b6d476cfa89f133babbe1b2f4f595bd8f4c476597f0a7c5f98e8ab4657ff7dc9
Jul 26 15:50:09.575930 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.575911    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pwcmg"
Jul 26 15:50:09.583977 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:09.583954    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda59a089c_67e4_455b_9936_af62b889eda9.slice/crio-eca41120bdd4e03266c7ee40feccdea3e950be7f68ba645a1e290f0f60a64551 WatchSource:0}: Error finding container eca41120bdd4e03266c7ee40feccdea3e950be7f68ba645a1e290f0f60a64551: Status 404 returned error can't find the container with id eca41120bdd4e03266c7ee40feccdea3e950be7f68ba645a1e290f0f60a64551
Jul 26 15:50:09.592499 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.592484    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-plqcw"
Jul 26 15:50:09.601410 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:09.601385    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda415587b_d9d6_4f2d_a37b_a49bb34aba81.slice/crio-85a3acdf19032f9dda72f1db99600be4aae936fe54f9029bd8dbc257b0fa3a73 WatchSource:0}: Error finding container 85a3acdf19032f9dda72f1db99600be4aae936fe54f9029bd8dbc257b0fa3a73: Status 404 returned error can't find the container with id 85a3acdf19032f9dda72f1db99600be4aae936fe54f9029bd8dbc257b0fa3a73
Jul 26 15:50:09.606794 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.606777    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kwjfd"
Jul 26 15:50:09.614806 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:09.614782    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31754f05_c0cf_4fa6_a3b5_f1afff0b7c99.slice/crio-03b2080437ee660574eae28f0d283a7321c24ffc239eb32e93e16cf4d27c3c2c WatchSource:0}: Error finding container 03b2080437ee660574eae28f0d283a7321c24ffc239eb32e93e16cf4d27c3c2c: Status 404 returned error can't find the container with id 03b2080437ee660574eae28f0d283a7321c24ffc239eb32e93e16cf4d27c3c2c
Jul 26 15:50:09.624347 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.624327    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zvsxm"
Jul 26 15:50:09.633302 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:09.633274    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6f459fa_098f_40e8_9086_bde5014c0807.slice/crio-1d6c341319bd2fe7295795a3b1f44244c68eefcd1306fb8407e12fd1532874f8 WatchSource:0}: Error finding container 1d6c341319bd2fe7295795a3b1f44244c68eefcd1306fb8407e12fd1532874f8: Status 404 returned error can't find the container with id 1d6c341319bd2fe7295795a3b1f44244c68eefcd1306fb8407e12fd1532874f8
Jul 26 15:50:09.639911 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.639889    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gnkxb"
Jul 26 15:50:09.644945 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.644927    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pj48f"
Jul 26 15:50:09.649684 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:09.649661    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaba3d036_e332_4b06_8379_7deb395345ba.slice/crio-bceeba69ffe6a53ab80c29a89461018615a70c9c1e50cf269852d77dba7e99b0 WatchSource:0}: Error finding container bceeba69ffe6a53ab80c29a89461018615a70c9c1e50cf269852d77dba7e99b0: Status 404 returned error can't find the container with id bceeba69ffe6a53ab80c29a89461018615a70c9c1e50cf269852d77dba7e99b0
Jul 26 15:50:09.650027 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.650010    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-t7zjp"
Jul 26 15:50:09.654655 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:09.654636    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c48fa94_94f6_43da_af02_7d63be16a9b8.slice/crio-8126ba9367d1df03b429a9e1abb9e96a3c5dd2a61c222efc6974bc1006668199 WatchSource:0}: Error finding container 8126ba9367d1df03b429a9e1abb9e96a3c5dd2a61c222efc6974bc1006668199: Status 404 returned error can't find the container with id 8126ba9367d1df03b429a9e1abb9e96a3c5dd2a61c222efc6974bc1006668199
Jul 26 15:50:09.660971 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:09.660949    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod705b8a13_4555_4820_8e91_e2ee6dbac16e.slice/crio-f38be6deb3b57c634fdb1925ff604273d8caf485d59888338c72837df8f03548 WatchSource:0}: Error finding container f38be6deb3b57c634fdb1925ff604273d8caf485d59888338c72837df8f03548: Status 404 returned error can't find the container with id f38be6deb3b57c634fdb1925ff604273d8caf485d59888338c72837df8f03548
Jul 26 15:50:09.666532 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.666515    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jltnr"
Jul 26 15:50:09.675784 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:09.675765    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod236e97e8_5df0_4a36_9513_a260891d2e9e.slice/crio-b94f88a7c1eb62469366e753adc48ef5cee72e71840e949d7350dda9237c1530 WatchSource:0}: Error finding container b94f88a7c1eb62469366e753adc48ef5cee72e71840e949d7350dda9237c1530: Status 404 returned error can't find the container with id b94f88a7c1eb62469366e753adc48ef5cee72e71840e949d7350dda9237c1530
Jul 26 15:50:09.959175 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.959085    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e62cc557-6267-4993-84a1-a888ced92345-original-pull-secret\") pod \"global-pull-secret-syncer-cxfp4\" (UID: \"e62cc557-6267-4993-84a1-a888ced92345\") " pod="kube-system/global-pull-secret-syncer-cxfp4"
Jul 26 15:50:09.959175 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:09.959156    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca110f78-ba25-4be3-9bed-60e5ef15e2c5-metrics-certs\") pod \"network-metrics-daemon-8gzcl\" (UID: \"ca110f78-ba25-4be3-9bed-60e5ef15e2c5\") " pod="openshift-multus/network-metrics-daemon-8gzcl"
Jul 26 15:50:09.959395 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:09.959264    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 26 15:50:09.959395 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:09.959341    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e62cc557-6267-4993-84a1-a888ced92345-original-pull-secret podName:e62cc557-6267-4993-84a1-a888ced92345 nodeName:}" failed. No retries permitted until 2026-07-26 15:50:10.959320365 +0000 UTC m=+3.063546157 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e62cc557-6267-4993-84a1-a888ced92345-original-pull-secret") pod "global-pull-secret-syncer-cxfp4" (UID: "e62cc557-6267-4993-84a1-a888ced92345") : object "kube-system"/"original-pull-secret" not registered
Jul 26 15:50:09.959395 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:09.959273    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 26 15:50:09.959522 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:09.959404    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca110f78-ba25-4be3-9bed-60e5ef15e2c5-metrics-certs podName:ca110f78-ba25-4be3-9bed-60e5ef15e2c5 nodeName:}" failed. No retries permitted until 2026-07-26 15:50:10.959382369 +0000 UTC m=+3.063608157 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca110f78-ba25-4be3-9bed-60e5ef15e2c5-metrics-certs") pod "network-metrics-daemon-8gzcl" (UID: "ca110f78-ba25-4be3-9bed-60e5ef15e2c5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 26 15:50:10.060181 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:10.060086    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmxk6\" (UniqueName: \"kubernetes.io/projected/0c90caab-1908-4071-a259-e0343960cbe2-kube-api-access-bmxk6\") pod \"network-check-target-7g75t\" (UID: \"0c90caab-1908-4071-a259-e0343960cbe2\") " pod="openshift-network-diagnostics/network-check-target-7g75t"
Jul 26 15:50:10.060353 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:10.060294    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 26 15:50:10.060353 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:10.060319    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 26 15:50:10.060353 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:10.060332    2563 projected.go:196] Error preparing data for projected volume kube-api-access-bmxk6 for pod openshift-network-diagnostics/network-check-target-7g75t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 26 15:50:10.060541 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:10.060394    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0c90caab-1908-4071-a259-e0343960cbe2-kube-api-access-bmxk6 podName:0c90caab-1908-4071-a259-e0343960cbe2 nodeName:}" failed. No retries permitted until 2026-07-26 15:50:11.060372042 +0000 UTC m=+3.164597833 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-bmxk6" (UniqueName: "kubernetes.io/projected/0c90caab-1908-4071-a259-e0343960cbe2-kube-api-access-bmxk6") pod "network-check-target-7g75t" (UID: "0c90caab-1908-4071-a259-e0343960cbe2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 26 15:50:10.125731 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:10.125529    2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 26 15:50:10.285860 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:10.285675    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-25 15:45:09 +0000 UTC" deadline="2028-01-17 18:19:10.236383531 +0000 UTC"
Jul 26 15:50:10.285860 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:10.285741    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12962h28m59.950646953s"
Jul 26 15:50:10.368665 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:10.368028    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cxfp4"
Jul 26 15:50:10.368665 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:10.368164    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cxfp4" podUID="e62cc557-6267-4993-84a1-a888ced92345"
Jul 26 15:50:10.400581 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:10.400517    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jltnr" event={"ID":"236e97e8-5df0-4a36-9513-a260891d2e9e","Type":"ContainerStarted","Data":"b94f88a7c1eb62469366e753adc48ef5cee72e71840e949d7350dda9237c1530"}
Jul 26 15:50:10.426856 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:10.422727    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gnkxb" event={"ID":"aba3d036-e332-4b06-8379-7deb395345ba","Type":"ContainerStarted","Data":"bceeba69ffe6a53ab80c29a89461018615a70c9c1e50cf269852d77dba7e99b0"}
Jul 26 15:50:10.441092 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:10.441051    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zvsxm" event={"ID":"a6f459fa-098f-40e8-9086-bde5014c0807","Type":"ContainerStarted","Data":"1d6c341319bd2fe7295795a3b1f44244c68eefcd1306fb8407e12fd1532874f8"}
Jul 26 15:50:10.445126 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:10.445032    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-204.ec2.internal" event={"ID":"5e83b5d2069ff1f9446be40c52eae0c3","Type":"ContainerStarted","Data":"86a6e4c1103c8322898f061ac7666ec7503408f4204e2876227a29129bda22b1"}
Jul 26 15:50:10.470814 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:10.470774    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-204.ec2.internal" event={"ID":"6db9e0d726c4428a7173af2b853a8bd3","Type":"ContainerStarted","Data":"031037d58cd9131311bbb8fb55a34c729ea77009b0ee2dbb01e5f45f57fc65f8"}
Jul 26 15:50:10.481970 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:10.481882    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-t7zjp" event={"ID":"705b8a13-4555-4820-8e91-e2ee6dbac16e","Type":"ContainerStarted","Data":"f38be6deb3b57c634fdb1925ff604273d8caf485d59888338c72837df8f03548"}
Jul 26 15:50:10.517039 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:10.516998    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pj48f" event={"ID":"0c48fa94-94f6-43da-af02-7d63be16a9b8","Type":"ContainerStarted","Data":"8126ba9367d1df03b429a9e1abb9e96a3c5dd2a61c222efc6974bc1006668199"}
Jul 26 15:50:10.533214 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:10.532886    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kwjfd" event={"ID":"31754f05-c0cf-4fa6-a3b5-f1afff0b7c99","Type":"ContainerStarted","Data":"03b2080437ee660574eae28f0d283a7321c24ffc239eb32e93e16cf4d27c3c2c"}
Jul 26 15:50:10.556527 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:10.556441    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-plqcw" event={"ID":"a415587b-d9d6-4f2d-a37b-a49bb34aba81","Type":"ContainerStarted","Data":"85a3acdf19032f9dda72f1db99600be4aae936fe54f9029bd8dbc257b0fa3a73"}
Jul 26 15:50:10.584512 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:10.584474    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pwcmg" event={"ID":"a59a089c-67e4-455b-9936-af62b889eda9","Type":"ContainerStarted","Data":"eca41120bdd4e03266c7ee40feccdea3e950be7f68ba645a1e290f0f60a64551"}
Jul 26 15:50:10.591383 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:10.591207    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-p8kcf" event={"ID":"e1f2fc7a-9737-422c-9c17-c8821dc48396","Type":"ContainerStarted","Data":"b6d476cfa89f133babbe1b2f4f595bd8f4c476597f0a7c5f98e8ab4657ff7dc9"}
Jul 26 15:50:10.627642 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:10.627608    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 26 15:50:10.966431 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:10.966345    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca110f78-ba25-4be3-9bed-60e5ef15e2c5-metrics-certs\") pod \"network-metrics-daemon-8gzcl\" (UID: \"ca110f78-ba25-4be3-9bed-60e5ef15e2c5\") " pod="openshift-multus/network-metrics-daemon-8gzcl"
Jul 26 15:50:10.966431 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:10.966421    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e62cc557-6267-4993-84a1-a888ced92345-original-pull-secret\") pod \"global-pull-secret-syncer-cxfp4\" (UID: \"e62cc557-6267-4993-84a1-a888ced92345\") " pod="kube-system/global-pull-secret-syncer-cxfp4"
Jul 26 15:50:10.966821 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:10.966560    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 26 15:50:10.966821 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:10.966640    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e62cc557-6267-4993-84a1-a888ced92345-original-pull-secret podName:e62cc557-6267-4993-84a1-a888ced92345 nodeName:}" failed. No retries permitted until 2026-07-26 15:50:12.966614053 +0000 UTC m=+5.070839841 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e62cc557-6267-4993-84a1-a888ced92345-original-pull-secret") pod "global-pull-secret-syncer-cxfp4" (UID: "e62cc557-6267-4993-84a1-a888ced92345") : object "kube-system"/"original-pull-secret" not registered
Jul 26 15:50:10.967068 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:10.967048    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 26 15:50:10.967147 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:10.967105    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca110f78-ba25-4be3-9bed-60e5ef15e2c5-metrics-certs podName:ca110f78-ba25-4be3-9bed-60e5ef15e2c5 nodeName:}" failed. No retries permitted until 2026-07-26 15:50:12.96709013 +0000 UTC m=+5.071315918 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca110f78-ba25-4be3-9bed-60e5ef15e2c5-metrics-certs") pod "network-metrics-daemon-8gzcl" (UID: "ca110f78-ba25-4be3-9bed-60e5ef15e2c5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 26 15:50:11.067207 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:11.067164    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmxk6\" (UniqueName: \"kubernetes.io/projected/0c90caab-1908-4071-a259-e0343960cbe2-kube-api-access-bmxk6\") pod \"network-check-target-7g75t\" (UID: \"0c90caab-1908-4071-a259-e0343960cbe2\") " pod="openshift-network-diagnostics/network-check-target-7g75t"
Jul 26 15:50:11.067481 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:11.067358    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 26 15:50:11.067481 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:11.067380    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 26 15:50:11.067481 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:11.067394    2563 projected.go:196] Error preparing data for projected volume kube-api-access-bmxk6 for pod openshift-network-diagnostics/network-check-target-7g75t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 26 15:50:11.067481 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:11.067455    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0c90caab-1908-4071-a259-e0343960cbe2-kube-api-access-bmxk6 podName:0c90caab-1908-4071-a259-e0343960cbe2 nodeName:}" failed. No retries permitted until 2026-07-26 15:50:13.067435758 +0000 UTC m=+5.171661550 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-bmxk6" (UniqueName: "kubernetes.io/projected/0c90caab-1908-4071-a259-e0343960cbe2-kube-api-access-bmxk6") pod "network-check-target-7g75t" (UID: "0c90caab-1908-4071-a259-e0343960cbe2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 26 15:50:11.286195 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:11.286087    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-25 15:45:09 +0000 UTC" deadline="2028-02-17 13:52:41.679272271 +0000 UTC"
Jul 26 15:50:11.286195 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:11.286144    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13702h2m30.393132767s"
Jul 26 15:50:11.367211 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:11.367117    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7g75t"
Jul 26 15:50:11.367478 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:11.367255    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7g75t" podUID="0c90caab-1908-4071-a259-e0343960cbe2"
Jul 26 15:50:11.367878 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:11.367674    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8gzcl"
Jul 26 15:50:11.367878 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:11.367798    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8gzcl" podUID="ca110f78-ba25-4be3-9bed-60e5ef15e2c5"
Jul 26 15:50:11.542125 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:11.542005    2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 26 15:50:12.090218 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:12.089448    2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 26 15:50:12.370656 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:12.370572    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cxfp4"
Jul 26 15:50:12.371157 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:12.370708    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cxfp4" podUID="e62cc557-6267-4993-84a1-a888ced92345"
Jul 26 15:50:12.984166 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:12.984125    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca110f78-ba25-4be3-9bed-60e5ef15e2c5-metrics-certs\") pod \"network-metrics-daemon-8gzcl\" (UID: \"ca110f78-ba25-4be3-9bed-60e5ef15e2c5\") " pod="openshift-multus/network-metrics-daemon-8gzcl"
Jul 26 15:50:12.984341 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:12.984215    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e62cc557-6267-4993-84a1-a888ced92345-original-pull-secret\") pod \"global-pull-secret-syncer-cxfp4\" (UID: \"e62cc557-6267-4993-84a1-a888ced92345\") " pod="kube-system/global-pull-secret-syncer-cxfp4"
Jul 26 15:50:12.984416 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:12.984351    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 26 15:50:12.984473 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:12.984418    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e62cc557-6267-4993-84a1-a888ced92345-original-pull-secret podName:e62cc557-6267-4993-84a1-a888ced92345 nodeName:}" failed. No retries permitted until 2026-07-26 15:50:16.984398371 +0000 UTC m=+9.088624180 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e62cc557-6267-4993-84a1-a888ced92345-original-pull-secret") pod "global-pull-secret-syncer-cxfp4" (UID: "e62cc557-6267-4993-84a1-a888ced92345") : object "kube-system"/"original-pull-secret" not registered
Jul 26 15:50:12.985258 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:12.984595    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 26 15:50:12.985258 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:12.984643    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca110f78-ba25-4be3-9bed-60e5ef15e2c5-metrics-certs podName:ca110f78-ba25-4be3-9bed-60e5ef15e2c5 nodeName:}" failed. No retries permitted until 2026-07-26 15:50:16.984628264 +0000 UTC m=+9.088854055 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca110f78-ba25-4be3-9bed-60e5ef15e2c5-metrics-certs") pod "network-metrics-daemon-8gzcl" (UID: "ca110f78-ba25-4be3-9bed-60e5ef15e2c5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 26 15:50:13.085651 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:13.085396    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmxk6\" (UniqueName: \"kubernetes.io/projected/0c90caab-1908-4071-a259-e0343960cbe2-kube-api-access-bmxk6\") pod \"network-check-target-7g75t\" (UID: \"0c90caab-1908-4071-a259-e0343960cbe2\") " pod="openshift-network-diagnostics/network-check-target-7g75t"
Jul 26 15:50:13.085651 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:13.085618    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 26 15:50:13.085651 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:13.085646    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 26 15:50:13.085948 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:13.085661    2563 projected.go:196] Error preparing data for projected volume kube-api-access-bmxk6 for pod openshift-network-diagnostics/network-check-target-7g75t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 26 15:50:13.085948 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:13.085727    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0c90caab-1908-4071-a259-e0343960cbe2-kube-api-access-bmxk6 podName:0c90caab-1908-4071-a259-e0343960cbe2 nodeName:}" failed. No retries permitted until 2026-07-26 15:50:17.085706494 +0000 UTC m=+9.189932297 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-bmxk6" (UniqueName: "kubernetes.io/projected/0c90caab-1908-4071-a259-e0343960cbe2-kube-api-access-bmxk6") pod "network-check-target-7g75t" (UID: "0c90caab-1908-4071-a259-e0343960cbe2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 26 15:50:13.368548 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:13.367711    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8gzcl"
Jul 26 15:50:13.368548 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:13.367884    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8gzcl" podUID="ca110f78-ba25-4be3-9bed-60e5ef15e2c5"
Jul 26 15:50:13.368548 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:13.368345    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7g75t"
Jul 26 15:50:13.368548 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:13.368444    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7g75t" podUID="0c90caab-1908-4071-a259-e0343960cbe2"
Jul 26 15:50:14.367207 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:14.367171    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cxfp4"
Jul 26 15:50:14.367817 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:14.367321    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cxfp4" podUID="e62cc557-6267-4993-84a1-a888ced92345"
Jul 26 15:50:15.367031 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:15.366989    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7g75t"
Jul 26 15:50:15.367227 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:15.366989    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8gzcl"
Jul 26 15:50:15.367227 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:15.367134    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7g75t" podUID="0c90caab-1908-4071-a259-e0343960cbe2"
Jul 26 15:50:15.367227 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:15.367174    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8gzcl" podUID="ca110f78-ba25-4be3-9bed-60e5ef15e2c5"
Jul 26 15:50:16.366704 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:16.366672    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cxfp4"
Jul 26 15:50:16.366872 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:16.366821    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cxfp4" podUID="e62cc557-6267-4993-84a1-a888ced92345"
Jul 26 15:50:17.018057 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:17.018004    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e62cc557-6267-4993-84a1-a888ced92345-original-pull-secret\") pod \"global-pull-secret-syncer-cxfp4\" (UID: \"e62cc557-6267-4993-84a1-a888ced92345\") " pod="kube-system/global-pull-secret-syncer-cxfp4"
Jul 26 15:50:17.018493 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:17.018073    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca110f78-ba25-4be3-9bed-60e5ef15e2c5-metrics-certs\") pod \"network-metrics-daemon-8gzcl\" (UID: \"ca110f78-ba25-4be3-9bed-60e5ef15e2c5\") " pod="openshift-multus/network-metrics-daemon-8gzcl"
Jul 26 15:50:17.018493 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:17.018226    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 26 15:50:17.018493 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:17.018285    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca110f78-ba25-4be3-9bed-60e5ef15e2c5-metrics-certs podName:ca110f78-ba25-4be3-9bed-60e5ef15e2c5 nodeName:}" failed. No retries permitted until 2026-07-26 15:50:25.018267322 +0000 UTC m=+17.122493108 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca110f78-ba25-4be3-9bed-60e5ef15e2c5-metrics-certs") pod "network-metrics-daemon-8gzcl" (UID: "ca110f78-ba25-4be3-9bed-60e5ef15e2c5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 26 15:50:17.018662 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:17.018531    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 26 15:50:17.018662 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:17.018601    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e62cc557-6267-4993-84a1-a888ced92345-original-pull-secret podName:e62cc557-6267-4993-84a1-a888ced92345 nodeName:}" failed. No retries permitted until 2026-07-26 15:50:25.018583393 +0000 UTC m=+17.122809192 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e62cc557-6267-4993-84a1-a888ced92345-original-pull-secret") pod "global-pull-secret-syncer-cxfp4" (UID: "e62cc557-6267-4993-84a1-a888ced92345") : object "kube-system"/"original-pull-secret" not registered
Jul 26 15:50:17.118846 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:17.118800    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmxk6\" (UniqueName: \"kubernetes.io/projected/0c90caab-1908-4071-a259-e0343960cbe2-kube-api-access-bmxk6\") pod \"network-check-target-7g75t\" (UID: \"0c90caab-1908-4071-a259-e0343960cbe2\") " pod="openshift-network-diagnostics/network-check-target-7g75t"
Jul 26 15:50:17.119025 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:17.119007    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 26 15:50:17.119096 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:17.119036    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 26 15:50:17.119096 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:17.119050    2563 projected.go:196] Error preparing data for projected volume kube-api-access-bmxk6 for pod openshift-network-diagnostics/network-check-target-7g75t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 26 15:50:17.119198 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:17.119116    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0c90caab-1908-4071-a259-e0343960cbe2-kube-api-access-bmxk6 podName:0c90caab-1908-4071-a259-e0343960cbe2 nodeName:}" failed. No retries permitted until 2026-07-26 15:50:25.119097605 +0000 UTC m=+17.223323414 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-bmxk6" (UniqueName: "kubernetes.io/projected/0c90caab-1908-4071-a259-e0343960cbe2-kube-api-access-bmxk6") pod "network-check-target-7g75t" (UID: "0c90caab-1908-4071-a259-e0343960cbe2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 26 15:50:17.367437 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:17.367346    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8gzcl"
Jul 26 15:50:17.367593 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:17.367501    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8gzcl" podUID="ca110f78-ba25-4be3-9bed-60e5ef15e2c5"
Jul 26 15:50:17.367593 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:17.367346    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7g75t"
Jul 26 15:50:17.367710 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:17.367590    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7g75t" podUID="0c90caab-1908-4071-a259-e0343960cbe2"
Jul 26 15:50:18.369347 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:18.367813    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cxfp4"
Jul 26 15:50:18.369347 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:18.367944    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cxfp4" podUID="e62cc557-6267-4993-84a1-a888ced92345"
Jul 26 15:50:19.367548 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:19.367461    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8gzcl"
Jul 26 15:50:19.367548 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:19.367483    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7g75t"
Jul 26 15:50:19.367791 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:19.367608    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8gzcl" podUID="ca110f78-ba25-4be3-9bed-60e5ef15e2c5"
Jul 26 15:50:19.367791 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:19.367735    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7g75t" podUID="0c90caab-1908-4071-a259-e0343960cbe2"
Jul 26 15:50:20.366744 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:20.366706    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cxfp4"
Jul 26 15:50:20.367211 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:20.366864    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cxfp4" podUID="e62cc557-6267-4993-84a1-a888ced92345"
Jul 26 15:50:21.367329 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:21.367301    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7g75t"
Jul 26 15:50:21.367329 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:21.367332    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8gzcl"
Jul 26 15:50:21.367704 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:21.367404    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7g75t" podUID="0c90caab-1908-4071-a259-e0343960cbe2"
Jul 26 15:50:21.367704 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:21.367449    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8gzcl" podUID="ca110f78-ba25-4be3-9bed-60e5ef15e2c5"
Jul 26 15:50:22.367595 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:22.367550    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cxfp4"
Jul 26 15:50:22.368064 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:22.367727    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cxfp4" podUID="e62cc557-6267-4993-84a1-a888ced92345"
Jul 26 15:50:23.366716 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:23.366682    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8gzcl"
Jul 26 15:50:23.366919 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:23.366723    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7g75t"
Jul 26 15:50:23.366919 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:23.366815    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8gzcl" podUID="ca110f78-ba25-4be3-9bed-60e5ef15e2c5"
Jul 26 15:50:23.367029 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:23.366946    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7g75t" podUID="0c90caab-1908-4071-a259-e0343960cbe2"
Jul 26 15:50:24.367322 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:24.367277    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cxfp4"
Jul 26 15:50:24.367750 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:24.367411    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cxfp4" podUID="e62cc557-6267-4993-84a1-a888ced92345"
Jul 26 15:50:25.080607 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:25.080572    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca110f78-ba25-4be3-9bed-60e5ef15e2c5-metrics-certs\") pod \"network-metrics-daemon-8gzcl\" (UID: \"ca110f78-ba25-4be3-9bed-60e5ef15e2c5\") " pod="openshift-multus/network-metrics-daemon-8gzcl"
Jul 26 15:50:25.080804 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:25.080633    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e62cc557-6267-4993-84a1-a888ced92345-original-pull-secret\") pod \"global-pull-secret-syncer-cxfp4\" (UID: \"e62cc557-6267-4993-84a1-a888ced92345\") " pod="kube-system/global-pull-secret-syncer-cxfp4"
Jul 26 15:50:25.080804 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:25.080716    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 26 15:50:25.080804 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:25.080718    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 26 15:50:25.080804 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:25.080784    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e62cc557-6267-4993-84a1-a888ced92345-original-pull-secret podName:e62cc557-6267-4993-84a1-a888ced92345 nodeName:}" failed. No retries permitted until 2026-07-26 15:50:41.080753933 +0000 UTC m=+33.184979720 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e62cc557-6267-4993-84a1-a888ced92345-original-pull-secret") pod "global-pull-secret-syncer-cxfp4" (UID: "e62cc557-6267-4993-84a1-a888ced92345") : object "kube-system"/"original-pull-secret" not registered
Jul 26 15:50:25.080804 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:25.080798    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca110f78-ba25-4be3-9bed-60e5ef15e2c5-metrics-certs podName:ca110f78-ba25-4be3-9bed-60e5ef15e2c5 nodeName:}" failed. No retries permitted until 2026-07-26 15:50:41.080791882 +0000 UTC m=+33.185017668 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca110f78-ba25-4be3-9bed-60e5ef15e2c5-metrics-certs") pod "network-metrics-daemon-8gzcl" (UID: "ca110f78-ba25-4be3-9bed-60e5ef15e2c5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 26 15:50:25.181665 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:25.181633    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmxk6\" (UniqueName: \"kubernetes.io/projected/0c90caab-1908-4071-a259-e0343960cbe2-kube-api-access-bmxk6\") pod \"network-check-target-7g75t\" (UID: \"0c90caab-1908-4071-a259-e0343960cbe2\") " pod="openshift-network-diagnostics/network-check-target-7g75t"
Jul 26 15:50:25.181865 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:25.181843    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 26 15:50:25.181915 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:25.181875    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 26 15:50:25.181915 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:25.181888    2563 projected.go:196] Error preparing data for projected volume kube-api-access-bmxk6 for pod openshift-network-diagnostics/network-check-target-7g75t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 26 15:50:25.181975 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:25.181954    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0c90caab-1908-4071-a259-e0343960cbe2-kube-api-access-bmxk6 podName:0c90caab-1908-4071-a259-e0343960cbe2 nodeName:}" failed. No retries permitted until 2026-07-26 15:50:41.181933789 +0000 UTC m=+33.286159582 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-bmxk6" (UniqueName: "kubernetes.io/projected/0c90caab-1908-4071-a259-e0343960cbe2-kube-api-access-bmxk6") pod "network-check-target-7g75t" (UID: "0c90caab-1908-4071-a259-e0343960cbe2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 26 15:50:25.366963 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:25.366848    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7g75t"
Jul 26 15:50:25.366963 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:25.366876    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8gzcl"
Jul 26 15:50:25.367213 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:25.366979    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7g75t" podUID="0c90caab-1908-4071-a259-e0343960cbe2"
Jul 26 15:50:25.367213 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:25.367137    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8gzcl" podUID="ca110f78-ba25-4be3-9bed-60e5ef15e2c5"
Jul 26 15:50:26.367411 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:26.367374    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cxfp4"
Jul 26 15:50:26.367817 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:26.367488    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cxfp4" podUID="e62cc557-6267-4993-84a1-a888ced92345"
Jul 26 15:50:27.367621 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:27.367585    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7g75t"
Jul 26 15:50:27.368065 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:27.367585    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8gzcl"
Jul 26 15:50:27.368065 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:27.367718    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7g75t" podUID="0c90caab-1908-4071-a259-e0343960cbe2"
Jul 26 15:50:27.368065 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:27.367879    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8gzcl" podUID="ca110f78-ba25-4be3-9bed-60e5ef15e2c5"
Jul 26 15:50:28.367659 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:28.367604    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cxfp4"
Jul 26 15:50:28.368123 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:28.367747    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cxfp4" podUID="e62cc557-6267-4993-84a1-a888ced92345"
Jul 26 15:50:29.366908 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:29.366743    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8gzcl"
Jul 26 15:50:29.366988 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:29.366789    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7g75t"
Jul 26 15:50:29.367050 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:29.367033    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8gzcl" podUID="ca110f78-ba25-4be3-9bed-60e5ef15e2c5"
Jul 26 15:50:29.367142 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:29.367124    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7g75t" podUID="0c90caab-1908-4071-a259-e0343960cbe2"
Jul 26 15:50:29.639964 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:29.639938    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kwjfd" event={"ID":"31754f05-c0cf-4fa6-a3b5-f1afff0b7c99","Type":"ContainerStarted","Data":"a1deb1bb8d315c484f6fc2dba67db6ad87bf907abfb6161c3a45be9374035e72"}
Jul 26 15:50:29.641901 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:29.641854    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pwcmg" event={"ID":"a59a089c-67e4-455b-9936-af62b889eda9","Type":"ContainerStarted","Data":"97972292f4a555376ecae58fac3d5209f3a73afb2a81b27c657a3fe7f75edb0a"}
Jul 26 15:50:29.644081 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:29.643963    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-204.ec2.internal" event={"ID":"6db9e0d726c4428a7173af2b853a8bd3","Type":"ContainerStarted","Data":"27f6417924cd7f4a88c1f83e902368868c7902b2dc6f22db3a4d83d35602c445"}
Jul 26 15:50:29.646145 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:29.646126    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-t7zjp" event={"ID":"705b8a13-4555-4820-8e91-e2ee6dbac16e","Type":"ContainerStarted","Data":"b7c11de5615d31b594cb02bd0c3a0675065fb1d2f94364948ea5ceb19a5b608c"}
Jul 26 15:50:29.658173 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:29.658116    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-pwcmg" podStartSLOduration=0.951693283 podStartE2EDuration="20.658101979s" totalImagesPullingTime="19.706408696s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 15:50:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:50:29.657690385 +0000 UTC m=+21.761916193" watchObservedRunningTime="2026-07-26 15:50:29.658101979 +0000 UTC m=+21.762327785"
Jul 26 15:50:29.670394 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:29.670356    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-204.ec2.internal" podStartSLOduration=20.670343735 podStartE2EDuration="20.670343735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 15:50:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:50:29.669680636 +0000 UTC m=+21.773906445" watchObservedRunningTime="2026-07-26 15:50:29.670343735 +0000 UTC m=+21.774569539"
Jul 26 15:50:29.687705 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:29.687644    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-t7zjp" podStartSLOduration=1.057575771 podStartE2EDuration="20.687623172s" totalImagesPullingTime="19.630047401s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 15:50:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:50:29.687029443 +0000 UTC m=+21.791255250" watchObservedRunningTime="2026-07-26 15:50:29.687623172 +0000 UTC m=+21.791848981"
Jul 26 15:50:30.367997 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:30.367751    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cxfp4"
Jul 26 15:50:30.368135 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:30.368045    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cxfp4" podUID="e62cc557-6267-4993-84a1-a888ced92345"
Jul 26 15:50:30.650133 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:30.650056    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pj48f" event={"ID":"0c48fa94-94f6-43da-af02-7d63be16a9b8","Type":"ContainerStarted","Data":"f1e161e642d65d0c057751fd22224ef3a4096a11049137e23aa588b962716168"}
Jul 26 15:50:30.653490 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:30.653459    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kwjfd" event={"ID":"31754f05-c0cf-4fa6-a3b5-f1afff0b7c99","Type":"ContainerStarted","Data":"88edaad1a33c06040b2a8ebf95c8d9f4aca282d7f319447dab71f1b361d52361"}
Jul 26 15:50:30.653600 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:30.653524    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kwjfd" event={"ID":"31754f05-c0cf-4fa6-a3b5-f1afff0b7c99","Type":"ContainerStarted","Data":"7df793023cd5e3add921b70596cb09e17956734f24443bf159b7f78c64fa74fc"}
Jul 26 15:50:30.653600 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:30.653568    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kwjfd" event={"ID":"31754f05-c0cf-4fa6-a3b5-f1afff0b7c99","Type":"ContainerStarted","Data":"94f07333e1347f38dd151a5e542704621327ff56cd9b5e99f32bad45a043daf4"}
Jul 26 15:50:30.653733 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:30.653615    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kwjfd" event={"ID":"31754f05-c0cf-4fa6-a3b5-f1afff0b7c99","Type":"ContainerStarted","Data":"740bb003232fc57b1aeddc6f4b792117f6da8710c899c15492ffe30f97c75203"}
Jul 26 15:50:30.653733 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:30.653665    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kwjfd" event={"ID":"31754f05-c0cf-4fa6-a3b5-f1afff0b7c99","Type":"ContainerStarted","Data":"121ad2ed32a8ce495b376b128313adbbaa4897f32f2077d3988c63079e2f54e8"}
Jul 26 15:50:30.655123 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:30.655096    2563 generic.go:356] "Generic (PLEG): container finished" podID="a415587b-d9d6-4f2d-a37b-a49bb34aba81" containerID="ea541584637db22c46a041db0651d24ab2e014af6b98c59001afdadf41b7a598" exitCode=0
Jul 26 15:50:30.655236 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:30.655203    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-plqcw" event={"ID":"a415587b-d9d6-4f2d-a37b-a49bb34aba81","Type":"ContainerDied","Data":"ea541584637db22c46a041db0651d24ab2e014af6b98c59001afdadf41b7a598"}
Jul 26 15:50:30.656936 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:30.656887    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-p8kcf" event={"ID":"e1f2fc7a-9737-422c-9c17-c8821dc48396","Type":"ContainerStarted","Data":"de28d508c605b03202961e87d6b13aeaf07ee242ad3325c55618f37dbb9131eb"}
Jul 26 15:50:30.658459 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:30.658438    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jltnr" event={"ID":"236e97e8-5df0-4a36-9513-a260891d2e9e","Type":"ContainerStarted","Data":"1fab8cb467be40fe4a5c2e7a7cc4f9f6e3173074c32578e0ac2e17f53c6deb89"}
Jul 26 15:50:30.659892 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:30.659868    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gnkxb" event={"ID":"aba3d036-e332-4b06-8379-7deb395345ba","Type":"ContainerStarted","Data":"94367ec1a550843d1069ec0f68e90cce25637404ceacf7c769474cd1f80ddae4"}
Jul 26 15:50:30.661525 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:30.661508    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zvsxm" event={"ID":"a6f459fa-098f-40e8-9086-bde5014c0807","Type":"ContainerStarted","Data":"b7702c8022f67e3178202ca8b8c99caf1ce3bc980c6e8562b407b206e8e2c024"}
Jul 26 15:50:30.663042 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:30.663019    2563 generic.go:356] "Generic (PLEG): container finished" podID="5e83b5d2069ff1f9446be40c52eae0c3" containerID="ab48908e7b608cc74ca082a056e3a54977791db922422f104f863cdac330ebdd" exitCode=0
Jul 26 15:50:30.663148 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:30.663126    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-204.ec2.internal" event={"ID":"5e83b5d2069ff1f9446be40c52eae0c3","Type":"ContainerDied","Data":"ab48908e7b608cc74ca082a056e3a54977791db922422f104f863cdac330ebdd"}
Jul 26 15:50:30.667279 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:30.667238    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-pj48f" podStartSLOduration=2.064915552 podStartE2EDuration="21.667227133s" totalImagesPullingTime="19.602311581s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:50:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:50:30.667196834 +0000 UTC m=+22.771422641" watchObservedRunningTime="2026-07-26 15:50:30.667227133 +0000 UTC m=+22.771452940"
Jul 26 15:50:30.682292 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:30.682260    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-jltnr" podStartSLOduration=2.1003421 podStartE2EDuration="21.682248142s" totalImagesPullingTime="19.581906042s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 15:50:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:50:30.682043887 +0000 UTC m=+22.786269706" watchObservedRunningTime="2026-07-26 15:50:30.682248142 +0000 UTC m=+22.786473950"
Jul 26 15:50:30.695404 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:30.695370    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-p8kcf" podStartSLOduration=10.762166526 podStartE2EDuration="22.695354947s" totalImagesPullingTime="11.933188421s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 15:50:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:50:30.694846315 +0000 UTC m=+22.799072124" watchObservedRunningTime="2026-07-26 15:50:30.695354947 +0000 UTC m=+22.799580754"
Jul 26 15:50:30.740096 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:30.739928    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-gnkxb" podStartSLOduration=2.132606897 podStartE2EDuration="21.739913642s" totalImagesPullingTime="19.607306745s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 15:50:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:50:30.739523381 +0000 UTC m=+22.843749202" watchObservedRunningTime="2026-07-26 15:50:30.739913642 +0000 UTC m=+22.844139450"
Jul 26 15:50:31.366862 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:31.366833    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8gzcl"
Jul 26 15:50:31.366993 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:31.366841    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7g75t"
Jul 26 15:50:31.366993 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:31.366948    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8gzcl" podUID="ca110f78-ba25-4be3-9bed-60e5ef15e2c5"
Jul 26 15:50:31.367131 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:31.367012    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7g75t" podUID="0c90caab-1908-4071-a259-e0343960cbe2"
Jul 26 15:50:31.413663 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:31.413635    2563 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 26 15:50:31.667821 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:31.667716    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zvsxm" event={"ID":"a6f459fa-098f-40e8-9086-bde5014c0807","Type":"ContainerStarted","Data":"5b14579eef286276bbb57ab70cdc0a6c64b4431b9adf6c54b2b95f32529152f4"}
Jul 26 15:50:31.669926 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:31.669898    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-204.ec2.internal" event={"ID":"5e83b5d2069ff1f9446be40c52eae0c3","Type":"ContainerStarted","Data":"3a106472397214654afebd4fb7330050d26a3d07a6d87f483cf2b834fc1bc669"}
Jul 26 15:50:31.684526 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:31.684479    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-204.ec2.internal" podStartSLOduration=22.684463779 podStartE2EDuration="22.684463779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 15:50:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:50:31.684174921 +0000 UTC m=+23.788400729" watchObservedRunningTime="2026-07-26 15:50:31.684463779 +0000 UTC m=+23.788689585"
Jul 26 15:50:32.307219 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:32.307091    2563 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-26T15:50:31.413655013Z","UUID":"abfe7804-9efd-4c85-bcdf-0b7eed009ca4","Handler":null,"Name":"","Endpoint":""}
Jul 26 15:50:32.308794 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:32.308756    2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 26 15:50:32.308912 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:32.308812    2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 26 15:50:32.367828 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:32.367677    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cxfp4"
Jul 26 15:50:32.367934 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:32.367893    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cxfp4" podUID="e62cc557-6267-4993-84a1-a888ced92345"
Jul 26 15:50:32.676648 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:32.676611    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kwjfd" event={"ID":"31754f05-c0cf-4fa6-a3b5-f1afff0b7c99","Type":"ContainerStarted","Data":"9e5ea57d5b0c5991b64d034d17ee230b628f40009a5f97d9b51f795c85dd897a"}
Jul 26 15:50:32.932313 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:32.932279    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-pj48f"
Jul 26 15:50:32.932857 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:32.932835    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-pj48f"
Jul 26 15:50:33.367298 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:33.367270    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8gzcl"
Jul 26 15:50:33.367514 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:33.367402    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8gzcl" podUID="ca110f78-ba25-4be3-9bed-60e5ef15e2c5"
Jul 26 15:50:33.367514 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:33.367463    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7g75t"
Jul 26 15:50:33.367628 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:33.367569    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7g75t" podUID="0c90caab-1908-4071-a259-e0343960cbe2"
Jul 26 15:50:33.681118 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:33.681030    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zvsxm" event={"ID":"a6f459fa-098f-40e8-9086-bde5014c0807","Type":"ContainerStarted","Data":"a915afe602a2d9d1e862f5ae28c853f45dde7313ba35888fb423eeabcd456566"}
Jul 26 15:50:33.681601 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:33.681254    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-pj48f"
Jul 26 15:50:33.681936 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:33.681915    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-pj48f"
Jul 26 15:50:33.711566 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:33.711522    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zvsxm" podStartSLOduration=2.652939745 podStartE2EDuration="24.711506297s" totalImagesPullingTime="22.058566552s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 15:50:09 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:50:33.697473747 +0000 UTC m=+25.801699556" watchObservedRunningTime="2026-07-26 15:50:33.711506297 +0000 UTC m=+25.815732108"
Jul 26 15:50:34.371119 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:34.370961    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cxfp4"
Jul 26 15:50:34.371119 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:34.371073    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cxfp4" podUID="e62cc557-6267-4993-84a1-a888ced92345"
Jul 26 15:50:34.688871 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:34.688798    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kwjfd" event={"ID":"31754f05-c0cf-4fa6-a3b5-f1afff0b7c99","Type":"ContainerStarted","Data":"e2cd1f526d50d1c90e46f8b0640704795b15b6efea377096c6c42fa378be9bae"}
Jul 26 15:50:34.714372 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:34.714219    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-kwjfd" podStartSLOduration=6.806865854 podStartE2EDuration="26.714204202s" totalImagesPullingTime="19.907338348s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 15:50:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:50:34.713656337 +0000 UTC m=+26.817882179" watchObservedRunningTime="2026-07-26 15:50:34.714204202 +0000 UTC m=+26.818430010"
Jul 26 15:50:35.367698 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:35.367510    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8gzcl"
Jul 26 15:50:35.367854 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:35.367522    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7g75t"
Jul 26 15:50:35.367854 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:35.367796    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8gzcl" podUID="ca110f78-ba25-4be3-9bed-60e5ef15e2c5"
Jul 26 15:50:35.367944 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:35.367856    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7g75t" podUID="0c90caab-1908-4071-a259-e0343960cbe2"
Jul 26 15:50:35.692393 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:35.692310    2563 generic.go:356] "Generic (PLEG): container finished" podID="a415587b-d9d6-4f2d-a37b-a49bb34aba81" containerID="d4d349042348d5e5ec1b8a20a89fab998ef238cff2b75ec5c9659d10fe061be7" exitCode=0
Jul 26 15:50:35.693090 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:35.692401    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-plqcw" event={"ID":"a415587b-d9d6-4f2d-a37b-a49bb34aba81","Type":"ContainerDied","Data":"d4d349042348d5e5ec1b8a20a89fab998ef238cff2b75ec5c9659d10fe061be7"}
Jul 26 15:50:35.693090 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:35.693043    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kwjfd"
Jul 26 15:50:35.693593 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:35.693110    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kwjfd"
Jul 26 15:50:35.693593 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:35.693160    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kwjfd"
Jul 26 15:50:35.707399 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:35.707376    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kwjfd"
Jul 26 15:50:35.707493 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:35.707466    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kwjfd"
Jul 26 15:50:36.368481 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:36.368449    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cxfp4"
Jul 26 15:50:36.368924 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:36.368672    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cxfp4" podUID="e62cc557-6267-4993-84a1-a888ced92345"
Jul 26 15:50:36.640533 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:36.640326    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8gzcl"]
Jul 26 15:50:36.640685 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:36.640653    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8gzcl"
Jul 26 15:50:36.640792 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:36.640756    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8gzcl" podUID="ca110f78-ba25-4be3-9bed-60e5ef15e2c5"
Jul 26 15:50:36.643708 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:36.643685    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cxfp4"]
Jul 26 15:50:36.644331 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:36.644288    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7g75t"]
Jul 26 15:50:36.644441 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:36.644385    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7g75t"
Jul 26 15:50:36.644532 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:36.644506    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7g75t" podUID="0c90caab-1908-4071-a259-e0343960cbe2"
Jul 26 15:50:36.694174 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:36.694151    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cxfp4"
Jul 26 15:50:36.694645 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:36.694246    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cxfp4" podUID="e62cc557-6267-4993-84a1-a888ced92345"
Jul 26 15:50:37.697744 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:37.697711    2563 generic.go:356] "Generic (PLEG): container finished" podID="a415587b-d9d6-4f2d-a37b-a49bb34aba81" containerID="54ef8e62f2f9eeffe832c88b32d4882b8238b50f43d633db22b1010e0f7313bd" exitCode=0
Jul 26 15:50:37.698164 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:37.697804    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-plqcw" event={"ID":"a415587b-d9d6-4f2d-a37b-a49bb34aba81","Type":"ContainerDied","Data":"54ef8e62f2f9eeffe832c88b32d4882b8238b50f43d633db22b1010e0f7313bd"}
Jul 26 15:50:38.367992 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:38.367962    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7g75t"
Jul 26 15:50:38.368171 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:38.368066    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7g75t" podUID="0c90caab-1908-4071-a259-e0343960cbe2"
Jul 26 15:50:38.368171 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:38.368136    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8gzcl"
Jul 26 15:50:38.368287 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:38.368226    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8gzcl" podUID="ca110f78-ba25-4be3-9bed-60e5ef15e2c5"
Jul 26 15:50:38.368287 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:38.368261    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cxfp4"
Jul 26 15:50:38.368392 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:38.368346    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cxfp4" podUID="e62cc557-6267-4993-84a1-a888ced92345"
Jul 26 15:50:39.704736 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:39.704650    2563 generic.go:356] "Generic (PLEG): container finished" podID="a415587b-d9d6-4f2d-a37b-a49bb34aba81" containerID="22e70d4d37bc431ea40a5e88f7f306f9872a0f4c53384a3d41ebe96a191b5cf2" exitCode=0
Jul 26 15:50:39.704736 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:39.704708    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-plqcw" event={"ID":"a415587b-d9d6-4f2d-a37b-a49bb34aba81","Type":"ContainerDied","Data":"22e70d4d37bc431ea40a5e88f7f306f9872a0f4c53384a3d41ebe96a191b5cf2"}
Jul 26 15:50:40.366795 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:40.366709    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cxfp4"
Jul 26 15:50:40.366989 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:40.366716    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8gzcl"
Jul 26 15:50:40.366989 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:40.366850    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cxfp4" podUID="e62cc557-6267-4993-84a1-a888ced92345"
Jul 26 15:50:40.366989 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:40.366859    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7g75t"
Jul 26 15:50:40.366989 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:40.366965    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8gzcl" podUID="ca110f78-ba25-4be3-9bed-60e5ef15e2c5"
Jul 26 15:50:40.367165 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:40.367049    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7g75t" podUID="0c90caab-1908-4071-a259-e0343960cbe2"
Jul 26 15:50:41.104832 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:41.104776    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e62cc557-6267-4993-84a1-a888ced92345-original-pull-secret\") pod \"global-pull-secret-syncer-cxfp4\" (UID: \"e62cc557-6267-4993-84a1-a888ced92345\") " pod="kube-system/global-pull-secret-syncer-cxfp4"
Jul 26 15:50:41.105251 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:41.104850    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca110f78-ba25-4be3-9bed-60e5ef15e2c5-metrics-certs\") pod \"network-metrics-daemon-8gzcl\" (UID: \"ca110f78-ba25-4be3-9bed-60e5ef15e2c5\") " pod="openshift-multus/network-metrics-daemon-8gzcl"
Jul 26 15:50:41.105251 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:41.104871    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 26 15:50:41.105251 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:41.104937    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e62cc557-6267-4993-84a1-a888ced92345-original-pull-secret podName:e62cc557-6267-4993-84a1-a888ced92345 nodeName:}" failed. No retries permitted until 2026-07-26 15:51:13.104920182 +0000 UTC m=+65.209145969 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e62cc557-6267-4993-84a1-a888ced92345-original-pull-secret") pod "global-pull-secret-syncer-cxfp4" (UID: "e62cc557-6267-4993-84a1-a888ced92345") : object "kube-system"/"original-pull-secret" not registered
Jul 26 15:50:41.105251 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:41.104954    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 26 15:50:41.105251 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:41.105003    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca110f78-ba25-4be3-9bed-60e5ef15e2c5-metrics-certs podName:ca110f78-ba25-4be3-9bed-60e5ef15e2c5 nodeName:}" failed. No retries permitted until 2026-07-26 15:51:13.10498833 +0000 UTC m=+65.209214120 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca110f78-ba25-4be3-9bed-60e5ef15e2c5-metrics-certs") pod "network-metrics-daemon-8gzcl" (UID: "ca110f78-ba25-4be3-9bed-60e5ef15e2c5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 26 15:50:41.205419 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:41.205377    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmxk6\" (UniqueName: \"kubernetes.io/projected/0c90caab-1908-4071-a259-e0343960cbe2-kube-api-access-bmxk6\") pod \"network-check-target-7g75t\" (UID: \"0c90caab-1908-4071-a259-e0343960cbe2\") " pod="openshift-network-diagnostics/network-check-target-7g75t"
Jul 26 15:50:41.205674 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:41.205617    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 26 15:50:41.205741 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:41.205682    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 26 15:50:41.205741 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:41.205716    2563 projected.go:196] Error preparing data for projected volume kube-api-access-bmxk6 for pod openshift-network-diagnostics/network-check-target-7g75t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 26 15:50:41.205858 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:41.205795    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0c90caab-1908-4071-a259-e0343960cbe2-kube-api-access-bmxk6 podName:0c90caab-1908-4071-a259-e0343960cbe2 nodeName:}" failed. No retries permitted until 2026-07-26 15:51:13.205776072 +0000 UTC m=+65.310001874 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-bmxk6" (UniqueName: "kubernetes.io/projected/0c90caab-1908-4071-a259-e0343960cbe2-kube-api-access-bmxk6") pod "network-check-target-7g75t" (UID: "0c90caab-1908-4071-a259-e0343960cbe2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 26 15:50:42.265144 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:42.265115    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-204.ec2.internal" event="NodeReady"
Jul 26 15:50:42.265571 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:42.265251    2563 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 26 15:50:42.306866 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:42.306829    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-9wvxg"]
Jul 26 15:50:42.310837 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:42.309631    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9wvxg"
Jul 26 15:50:42.310837 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:42.310334    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-9mkmv"]
Jul 26 15:50:42.312743 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:42.312717    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9mkmv"
Jul 26 15:50:42.312888 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:42.312812    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 26 15:50:42.312888 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:42.312812    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 26 15:50:42.312888 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:42.312873    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cx8dk\""
Jul 26 15:50:42.315064 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:42.315046    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 26 15:50:42.315194 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:42.315179    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-69pk7\""
Jul 26 15:50:42.315273 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:42.315192    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 26 15:50:42.315273 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:42.315202    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 26 15:50:42.317757 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:42.317738    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9wvxg"]
Jul 26 15:50:42.320916 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:42.320865    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9mkmv"]
Jul 26 15:50:42.366945 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:42.366914    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7g75t"
Jul 26 15:50:42.367126 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:42.366929    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8gzcl"
Jul 26 15:50:42.367126 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:42.367037    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cxfp4"
Jul 26 15:50:42.369279 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:42.369255    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 26 15:50:42.369450 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:42.369266    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 26 15:50:42.369450 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:42.369313    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 26 15:50:42.369450 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:42.369321    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-c5h9x\""
Jul 26 15:50:42.369450 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:42.369328    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-r7gjf\""
Jul 26 15:50:42.369682 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:42.369605    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 26 15:50:42.414531 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:42.414501    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c34afa8-d2eb-4651-8464-4a56f7605447-metrics-tls\") pod \"dns-default-9wvxg\" (UID: \"7c34afa8-d2eb-4651-8464-4a56f7605447\") " pod="openshift-dns/dns-default-9wvxg"
Jul 26 15:50:42.414695 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:42.414544    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c34afa8-d2eb-4651-8464-4a56f7605447-tmp-dir\") pod \"dns-default-9wvxg\" (UID: \"7c34afa8-d2eb-4651-8464-4a56f7605447\") " pod="openshift-dns/dns-default-9wvxg"
Jul 26 15:50:42.414695 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:42.414574    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5xkv\" (UniqueName: \"kubernetes.io/projected/7c34afa8-d2eb-4651-8464-4a56f7605447-kube-api-access-q5xkv\") pod \"dns-default-9wvxg\" (UID: \"7c34afa8-d2eb-4651-8464-4a56f7605447\") " pod="openshift-dns/dns-default-9wvxg"
Jul 26 15:50:42.414695 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:42.414627    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/214a33cc-fbe9-4dd9-8f42-14e737ac0341-cert\") pod \"ingress-canary-9mkmv\" (UID: \"214a33cc-fbe9-4dd9-8f42-14e737ac0341\") " pod="openshift-ingress-canary/ingress-canary-9mkmv"
Jul 26 15:50:42.414695 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:42.414649    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br669\" (UniqueName: \"kubernetes.io/projected/214a33cc-fbe9-4dd9-8f42-14e737ac0341-kube-api-access-br669\") pod \"ingress-canary-9mkmv\" (UID: \"214a33cc-fbe9-4dd9-8f42-14e737ac0341\") " pod="openshift-ingress-canary/ingress-canary-9mkmv"
Jul 26 15:50:42.414921 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:42.414704    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7c34afa8-d2eb-4651-8464-4a56f7605447-config-volume\") pod \"dns-default-9wvxg\" (UID: \"7c34afa8-d2eb-4651-8464-4a56f7605447\") " pod="openshift-dns/dns-default-9wvxg"
Jul 26 15:50:42.515958 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:42.515872    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7c34afa8-d2eb-4651-8464-4a56f7605447-config-volume\") pod \"dns-default-9wvxg\" (UID: \"7c34afa8-d2eb-4651-8464-4a56f7605447\") " pod="openshift-dns/dns-default-9wvxg"
Jul 26 15:50:42.515958 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:42.515939    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c34afa8-d2eb-4651-8464-4a56f7605447-metrics-tls\") pod \"dns-default-9wvxg\" (UID: \"7c34afa8-d2eb-4651-8464-4a56f7605447\") " pod="openshift-dns/dns-default-9wvxg"
Jul 26 15:50:42.516167 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:42.516002    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c34afa8-d2eb-4651-8464-4a56f7605447-tmp-dir\") pod \"dns-default-9wvxg\" (UID: \"7c34afa8-d2eb-4651-8464-4a56f7605447\") " pod="openshift-dns/dns-default-9wvxg"
Jul 26 15:50:42.516167 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:42.516028    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5xkv\" (UniqueName: \"kubernetes.io/projected/7c34afa8-d2eb-4651-8464-4a56f7605447-kube-api-access-q5xkv\") pod \"dns-default-9wvxg\" (UID: \"7c34afa8-d2eb-4651-8464-4a56f7605447\") " pod="openshift-dns/dns-default-9wvxg"
Jul 26 15:50:42.516167 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:42.516059    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/214a33cc-fbe9-4dd9-8f42-14e737ac0341-cert\") pod \"ingress-canary-9mkmv\" (UID: \"214a33cc-fbe9-4dd9-8f42-14e737ac0341\") " pod="openshift-ingress-canary/ingress-canary-9mkmv"
Jul 26 15:50:42.516167 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:42.516067    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 26 15:50:42.516167 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:42.516083    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br669\" (UniqueName: \"kubernetes.io/projected/214a33cc-fbe9-4dd9-8f42-14e737ac0341-kube-api-access-br669\") pod \"ingress-canary-9mkmv\" (UID: \"214a33cc-fbe9-4dd9-8f42-14e737ac0341\") " pod="openshift-ingress-canary/ingress-canary-9mkmv"
Jul 26 15:50:42.516167 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:42.516150    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c34afa8-d2eb-4651-8464-4a56f7605447-metrics-tls podName:7c34afa8-d2eb-4651-8464-4a56f7605447 nodeName:}" failed. No retries permitted until 2026-07-26 15:50:43.016127761 +0000 UTC m=+35.120353564 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7c34afa8-d2eb-4651-8464-4a56f7605447-metrics-tls") pod "dns-default-9wvxg" (UID: "7c34afa8-d2eb-4651-8464-4a56f7605447") : secret "dns-default-metrics-tls" not found
Jul 26 15:50:42.516518 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:42.516496    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7c34afa8-d2eb-4651-8464-4a56f7605447-config-volume\") pod \"dns-default-9wvxg\" (UID: \"7c34afa8-d2eb-4651-8464-4a56f7605447\") " pod="openshift-dns/dns-default-9wvxg"
Jul 26 15:50:42.516582 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:42.516549    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c34afa8-d2eb-4651-8464-4a56f7605447-tmp-dir\") pod \"dns-default-9wvxg\" (UID: \"7c34afa8-d2eb-4651-8464-4a56f7605447\") " pod="openshift-dns/dns-default-9wvxg"
Jul 26 15:50:42.521103 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:42.521082    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/214a33cc-fbe9-4dd9-8f42-14e737ac0341-cert\") pod \"ingress-canary-9mkmv\" (UID: \"214a33cc-fbe9-4dd9-8f42-14e737ac0341\") " pod="openshift-ingress-canary/ingress-canary-9mkmv"
Jul 26 15:50:42.524456 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:42.524199    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5xkv\" (UniqueName: \"kubernetes.io/projected/7c34afa8-d2eb-4651-8464-4a56f7605447-kube-api-access-q5xkv\") pod \"dns-default-9wvxg\" (UID: \"7c34afa8-d2eb-4651-8464-4a56f7605447\") " pod="openshift-dns/dns-default-9wvxg"
Jul 26 15:50:42.524456 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:42.524255    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-br669\" (UniqueName: \"kubernetes.io/projected/214a33cc-fbe9-4dd9-8f42-14e737ac0341-kube-api-access-br669\") pod \"ingress-canary-9mkmv\" (UID: \"214a33cc-fbe9-4dd9-8f42-14e737ac0341\") " pod="openshift-ingress-canary/ingress-canary-9mkmv"
Jul 26 15:50:42.629527 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:42.629486    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9mkmv"
Jul 26 15:50:42.758020 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:42.757978    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:50:42.839956 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:42.839881    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:50:42.929929 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:42.929870    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:50:42.991989 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:42.991952    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:50:43.021406 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:43.021364    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c34afa8-d2eb-4651-8464-4a56f7605447-metrics-tls\") pod \"dns-default-9wvxg\" (UID: \"7c34afa8-d2eb-4651-8464-4a56f7605447\") " pod="openshift-dns/dns-default-9wvxg"
Jul 26 15:50:43.021583 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:43.021542    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 26 15:50:43.021649 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:43.021626    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c34afa8-d2eb-4651-8464-4a56f7605447-metrics-tls podName:7c34afa8-d2eb-4651-8464-4a56f7605447 nodeName:}" failed. No retries permitted until 2026-07-26 15:50:44.021606788 +0000 UTC m=+36.125832581 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7c34afa8-d2eb-4651-8464-4a56f7605447-metrics-tls") pod "dns-default-9wvxg" (UID: "7c34afa8-d2eb-4651-8464-4a56f7605447") : secret "dns-default-metrics-tls" not found
Jul 26 15:50:43.524825 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:43.524791    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9mkmv"]
Jul 26 15:50:43.528675 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:43.528643    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod214a33cc_fbe9_4dd9_8f42_14e737ac0341.slice/crio-0c2f44463449d86a21e9a258fa453f4175d31f912eb3d2eb1dd9426c6d3c124f WatchSource:0}: Error finding container 0c2f44463449d86a21e9a258fa453f4175d31f912eb3d2eb1dd9426c6d3c124f: Status 404 returned error can't find the container with id 0c2f44463449d86a21e9a258fa453f4175d31f912eb3d2eb1dd9426c6d3c124f
Jul 26 15:50:43.714976 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:43.714899    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9mkmv" event={"ID":"214a33cc-fbe9-4dd9-8f42-14e737ac0341","Type":"ContainerStarted","Data":"0c2f44463449d86a21e9a258fa453f4175d31f912eb3d2eb1dd9426c6d3c124f"}
Jul 26 15:50:43.717583 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:43.717564    2563 generic.go:356] "Generic (PLEG): container finished" podID="a415587b-d9d6-4f2d-a37b-a49bb34aba81" containerID="92c4451c2821ffe80c87555122c61b4fc693482393e6e7630b5c90db3b38e68c" exitCode=0
Jul 26 15:50:43.717680 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:43.717595    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-plqcw" event={"ID":"a415587b-d9d6-4f2d-a37b-a49bb34aba81","Type":"ContainerDied","Data":"92c4451c2821ffe80c87555122c61b4fc693482393e6e7630b5c90db3b38e68c"}
Jul 26 15:50:43.985689 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:43.985656    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:50:44.028503 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:44.028445    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c34afa8-d2eb-4651-8464-4a56f7605447-metrics-tls\") pod \"dns-default-9wvxg\" (UID: \"7c34afa8-d2eb-4651-8464-4a56f7605447\") " pod="openshift-dns/dns-default-9wvxg"
Jul 26 15:50:44.028642 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:44.028616    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 26 15:50:44.028700 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:44.028688    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c34afa8-d2eb-4651-8464-4a56f7605447-metrics-tls podName:7c34afa8-d2eb-4651-8464-4a56f7605447 nodeName:}" failed. No retries permitted until 2026-07-26 15:50:46.028668009 +0000 UTC m=+38.132893815 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7c34afa8-d2eb-4651-8464-4a56f7605447-metrics-tls") pod "dns-default-9wvxg" (UID: "7c34afa8-d2eb-4651-8464-4a56f7605447") : secret "dns-default-metrics-tls" not found
Jul 26 15:50:44.088757 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:44.088711    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:50:44.723776 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:44.723729    2563 generic.go:356] "Generic (PLEG): container finished" podID="a415587b-d9d6-4f2d-a37b-a49bb34aba81" containerID="1e79d54447bff6e92c17b47e390f3bbc67e169e46c9a01651b361ff30a9f5485" exitCode=0
Jul 26 15:50:44.724185 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:44.723799    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-plqcw" event={"ID":"a415587b-d9d6-4f2d-a37b-a49bb34aba81","Type":"ContainerDied","Data":"1e79d54447bff6e92c17b47e390f3bbc67e169e46c9a01651b361ff30a9f5485"}
Jul 26 15:50:45.728986 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:45.728801    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-plqcw" event={"ID":"a415587b-d9d6-4f2d-a37b-a49bb34aba81","Type":"ContainerStarted","Data":"7164a792d3458a63e4fb4ad695c5b42ec8aa932d4a39f9855e28b60d06378429"}
Jul 26 15:50:45.750507 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:45.750462    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-plqcw" podStartSLOduration=8.285643017 podStartE2EDuration="37.750447913s" totalImagesPullingTime="29.464804896s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 15:50:08 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:50:45.750336119 +0000 UTC m=+37.854561927" watchObservedRunningTime="2026-07-26 15:50:45.750447913 +0000 UTC m=+37.854673721"
Jul 26 15:50:45.870728 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:45.870701    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:50:45.970358 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:45.970325    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:50:46.043470 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:46.043358    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c34afa8-d2eb-4651-8464-4a56f7605447-metrics-tls\") pod \"dns-default-9wvxg\" (UID: \"7c34afa8-d2eb-4651-8464-4a56f7605447\") " pod="openshift-dns/dns-default-9wvxg"
Jul 26 15:50:46.043605 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:46.043516    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 26 15:50:46.043605 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:46.043584    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c34afa8-d2eb-4651-8464-4a56f7605447-metrics-tls podName:7c34afa8-d2eb-4651-8464-4a56f7605447 nodeName:}" failed. No retries permitted until 2026-07-26 15:50:50.043568069 +0000 UTC m=+42.147793858 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7c34afa8-d2eb-4651-8464-4a56f7605447-metrics-tls") pod "dns-default-9wvxg" (UID: "7c34afa8-d2eb-4651-8464-4a56f7605447") : secret "dns-default-metrics-tls" not found
Jul 26 15:50:46.576193 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:46.576162    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:50:46.641941 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:46.641911    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:50:46.732472 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:46.732432    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9mkmv" event={"ID":"214a33cc-fbe9-4dd9-8f42-14e737ac0341","Type":"ContainerStarted","Data":"00457b4964ac0d5285443b870d7ec9d08476feb91b9ed63a59ec66994913f882"}
Jul 26 15:50:46.748049 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:46.748003    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-9mkmv" podStartSLOduration=2.481513649 podStartE2EDuration="4.7479882s" totalImagesPullingTime="2.266474551s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:50:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:50:46.746808217 +0000 UTC m=+38.851034024" watchObservedRunningTime="2026-07-26 15:50:46.7479882 +0000 UTC m=+38.852214008"
Jul 26 15:50:46.798004 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:46.797977    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:50:46.853815 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:46.853720    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:50:47.497314 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:47.497284    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:50:47.596268 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:47.596242    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:50:47.867952 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:47.867878    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:50:47.893340 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:47.893310    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-6fgd7"]
Jul 26 15:50:47.911816 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:47.911787    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-6fgd7"]
Jul 26 15:50:47.911945 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:47.911903    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-6fgd7"
Jul 26 15:50:47.914324 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:47.914304    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 26 15:50:47.914324 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:47.914331    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 26 15:50:47.914962 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:47.914945    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-8psck\""
Jul 26 15:50:47.963038 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:47.963006    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:50:48.039661 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:48.039628    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:50:48.058736 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:48.058705    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcp58\" (UniqueName: \"kubernetes.io/projected/d572adcb-06e0-4f95-9de7-8d72b2428789-kube-api-access-mcp58\") pod \"migrator-6865c9967-6fgd7\" (UID: \"d572adcb-06e0-4f95-9de7-8d72b2428789\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-6fgd7"
Jul 26 15:50:48.159491 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:48.159416    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcp58\" (UniqueName: \"kubernetes.io/projected/d572adcb-06e0-4f95-9de7-8d72b2428789-kube-api-access-mcp58\") pod \"migrator-6865c9967-6fgd7\" (UID: \"d572adcb-06e0-4f95-9de7-8d72b2428789\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-6fgd7"
Jul 26 15:50:48.166919 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:48.166888    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:50:48.169423 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:48.169403    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcp58\" (UniqueName: \"kubernetes.io/projected/d572adcb-06e0-4f95-9de7-8d72b2428789-kube-api-access-mcp58\") pod \"migrator-6865c9967-6fgd7\" (UID: \"d572adcb-06e0-4f95-9de7-8d72b2428789\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-6fgd7"
Jul 26 15:50:48.220165 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:48.220131    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-6fgd7"
Jul 26 15:50:48.265445 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:48.265415    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:50:48.395776 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:48.395734    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-6fgd7"]
Jul 26 15:50:48.398494 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:48.398467    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd572adcb_06e0_4f95_9de7_8d72b2428789.slice/crio-a56c8536f784bfba374ad37f40ff09e079738de51520c67312fa859f2c748140 WatchSource:0}: Error finding container a56c8536f784bfba374ad37f40ff09e079738de51520c67312fa859f2c748140: Status 404 returned error can't find the container with id a56c8536f784bfba374ad37f40ff09e079738de51520c67312fa859f2c748140
Jul 26 15:50:48.738227 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:48.738191    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-6fgd7" event={"ID":"d572adcb-06e0-4f95-9de7-8d72b2428789","Type":"ContainerStarted","Data":"a56c8536f784bfba374ad37f40ff09e079738de51520c67312fa859f2c748140"}
Jul 26 15:50:49.804468 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:49.804435    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-lwn5k"]
Jul 26 15:50:49.826886 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:49.826857    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-lwn5k"]
Jul 26 15:50:49.827036 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:49.826960    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-lwn5k"
Jul 26 15:50:49.830117 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:49.830098    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 26 15:50:49.830225 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:49.830099    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-2qjdd\""
Jul 26 15:50:49.830975 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:49.830957    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 26 15:50:49.831083 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:49.830986    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 26 15:50:49.831083 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:49.831015    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 26 15:50:49.835351 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:49.835334    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:50:49.892642 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:49.892611    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:50:49.976475 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:49.976437    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4810cc00-bdf8-491c-b3bb-a192c326e679-signing-key\") pod \"service-ca-658d7b7c78-lwn5k\" (UID: \"4810cc00-bdf8-491c-b3bb-a192c326e679\") " pod="openshift-service-ca/service-ca-658d7b7c78-lwn5k"
Jul 26 15:50:49.976634 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:49.976482    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4810cc00-bdf8-491c-b3bb-a192c326e679-signing-cabundle\") pod \"service-ca-658d7b7c78-lwn5k\" (UID: \"4810cc00-bdf8-491c-b3bb-a192c326e679\") " pod="openshift-service-ca/service-ca-658d7b7c78-lwn5k"
Jul 26 15:50:49.976634 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:49.976499    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq5c6\" (UniqueName: \"kubernetes.io/projected/4810cc00-bdf8-491c-b3bb-a192c326e679-kube-api-access-cq5c6\") pod \"service-ca-658d7b7c78-lwn5k\" (UID: \"4810cc00-bdf8-491c-b3bb-a192c326e679\") " pod="openshift-service-ca/service-ca-658d7b7c78-lwn5k"
Jul 26 15:50:50.077735 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:50.077708    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4810cc00-bdf8-491c-b3bb-a192c326e679-signing-key\") pod \"service-ca-658d7b7c78-lwn5k\" (UID: \"4810cc00-bdf8-491c-b3bb-a192c326e679\") " pod="openshift-service-ca/service-ca-658d7b7c78-lwn5k"
Jul 26 15:50:50.077872 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:50.077755    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4810cc00-bdf8-491c-b3bb-a192c326e679-signing-cabundle\") pod \"service-ca-658d7b7c78-lwn5k\" (UID: \"4810cc00-bdf8-491c-b3bb-a192c326e679\") " pod="openshift-service-ca/service-ca-658d7b7c78-lwn5k"
Jul 26 15:50:50.077872 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:50.077793    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq5c6\" (UniqueName: \"kubernetes.io/projected/4810cc00-bdf8-491c-b3bb-a192c326e679-kube-api-access-cq5c6\") pod \"service-ca-658d7b7c78-lwn5k\" (UID: \"4810cc00-bdf8-491c-b3bb-a192c326e679\") " pod="openshift-service-ca/service-ca-658d7b7c78-lwn5k"
Jul 26 15:50:50.077960 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:50.077889    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c34afa8-d2eb-4651-8464-4a56f7605447-metrics-tls\") pod \"dns-default-9wvxg\" (UID: \"7c34afa8-d2eb-4651-8464-4a56f7605447\") " pod="openshift-dns/dns-default-9wvxg"
Jul 26 15:50:50.078007 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:50.077965    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 26 15:50:50.078052 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:50:50.078027    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c34afa8-d2eb-4651-8464-4a56f7605447-metrics-tls podName:7c34afa8-d2eb-4651-8464-4a56f7605447 nodeName:}" failed. No retries permitted until 2026-07-26 15:50:58.07800824 +0000 UTC m=+50.182234039 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7c34afa8-d2eb-4651-8464-4a56f7605447-metrics-tls") pod "dns-default-9wvxg" (UID: "7c34afa8-d2eb-4651-8464-4a56f7605447") : secret "dns-default-metrics-tls" not found
Jul 26 15:50:50.078517 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:50.078498    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4810cc00-bdf8-491c-b3bb-a192c326e679-signing-cabundle\") pod \"service-ca-658d7b7c78-lwn5k\" (UID: \"4810cc00-bdf8-491c-b3bb-a192c326e679\") " pod="openshift-service-ca/service-ca-658d7b7c78-lwn5k"
Jul 26 15:50:50.080230 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:50.080204    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4810cc00-bdf8-491c-b3bb-a192c326e679-signing-key\") pod \"service-ca-658d7b7c78-lwn5k\" (UID: \"4810cc00-bdf8-491c-b3bb-a192c326e679\") " pod="openshift-service-ca/service-ca-658d7b7c78-lwn5k"
Jul 26 15:50:50.086702 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:50.086682    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq5c6\" (UniqueName: \"kubernetes.io/projected/4810cc00-bdf8-491c-b3bb-a192c326e679-kube-api-access-cq5c6\") pod \"service-ca-658d7b7c78-lwn5k\" (UID: \"4810cc00-bdf8-491c-b3bb-a192c326e679\") " pod="openshift-service-ca/service-ca-658d7b7c78-lwn5k"
Jul 26 15:50:50.136125 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:50.136100    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-lwn5k"
Jul 26 15:50:50.281601 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:50.281567    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-lwn5k"]
Jul 26 15:50:50.284703 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:50.284675    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4810cc00_bdf8_491c_b3bb_a192c326e679.slice/crio-aefc2d3ed14fc3c4fbe1bb355757e44411308a2b85669a7a999252f7b85f9a5d WatchSource:0}: Error finding container aefc2d3ed14fc3c4fbe1bb355757e44411308a2b85669a7a999252f7b85f9a5d: Status 404 returned error can't find the container with id aefc2d3ed14fc3c4fbe1bb355757e44411308a2b85669a7a999252f7b85f9a5d
Jul 26 15:50:50.706730 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:50.706668    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:50:50.744514 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:50.744467    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-6fgd7" event={"ID":"d572adcb-06e0-4f95-9de7-8d72b2428789","Type":"ContainerStarted","Data":"30d913d581b245daa3bff9c362e9ed3d7d610a8f01d69878131e5d54bb8f169c"}
Jul 26 15:50:50.744691 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:50.744533    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-6fgd7" event={"ID":"d572adcb-06e0-4f95-9de7-8d72b2428789","Type":"ContainerStarted","Data":"3764a0a61ffd1ffe6584f2fac55b36cd2ee676017ed80bc8a1661d203ce60c67"}
Jul 26 15:50:50.745599 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:50.745571    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-lwn5k" event={"ID":"4810cc00-bdf8-491c-b3bb-a192c326e679","Type":"ContainerStarted","Data":"aefc2d3ed14fc3c4fbe1bb355757e44411308a2b85669a7a999252f7b85f9a5d"}
Jul 26 15:50:50.762023 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:50.761981    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-6fgd7" podStartSLOduration=2.203529871 podStartE2EDuration="3.761966458s" totalImagesPullingTime="1.558436587s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:50:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:50:50.761549502 +0000 UTC m=+42.865775349" watchObservedRunningTime="2026-07-26 15:50:50.761966458 +0000 UTC m=+42.866192266"
Jul 26 15:50:52.752096 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:52.752059    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-lwn5k" event={"ID":"4810cc00-bdf8-491c-b3bb-a192c326e679","Type":"ContainerStarted","Data":"323495ec7bc360cb3b23ab3a133a316b15fc18361fcfaf925a3aa3e2a250ccad"}
Jul 26 15:50:53.664927 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:53.664894    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:50:54.146463 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:54.146420    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:50:54.276912 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:54.276872    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:50:54.340233 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:54.340150    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:50:54.429989 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:54.429864    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:50:54.530352 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:54.530309    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:50:55.038029 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:55.037999    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:50:55.440282 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:55.440201    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:50:56.238069 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:56.238032    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:50:56.635006 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:56.634923    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:50:56.833901 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:56.833862    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:50:57.025754 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:57.025725    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:50:57.081941 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:57.081912    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:50:57.436362 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:57.436281    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:50:57.832082 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:57.832050    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:50:58.035270 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:58.035238    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:50:58.140689 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:58.140612    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c34afa8-d2eb-4651-8464-4a56f7605447-metrics-tls\") pod \"dns-default-9wvxg\" (UID: \"7c34afa8-d2eb-4651-8464-4a56f7605447\") " pod="openshift-dns/dns-default-9wvxg"
Jul 26 15:50:58.142918 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:58.142899    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c34afa8-d2eb-4651-8464-4a56f7605447-metrics-tls\") pod \"dns-default-9wvxg\" (UID: \"7c34afa8-d2eb-4651-8464-4a56f7605447\") " pod="openshift-dns/dns-default-9wvxg"
Jul 26 15:50:58.224165 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:58.224121    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9wvxg"
Jul 26 15:50:58.359749 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:58.359718    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:50:58.363541 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:58.363498    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-lwn5k" podStartSLOduration=7.330144297 podStartE2EDuration="9.363482583s" totalImagesPullingTime="2.033338286s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:50:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:50:52.767657346 +0000 UTC m=+44.871883157" watchObservedRunningTime="2026-07-26 15:50:58.363482583 +0000 UTC m=+50.467708383"
Jul 26 15:50:58.363893 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:58.363875    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9wvxg"]
Jul 26 15:50:58.368379 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:50:58.368352    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c34afa8_d2eb_4651_8464_4a56f7605447.slice/crio-bbabb62d3880ecda518c3b1346f775fed5497598b695590383f0b308a1a66cd6 WatchSource:0}: Error finding container bbabb62d3880ecda518c3b1346f775fed5497598b695590383f0b308a1a66cd6: Status 404 returned error can't find the container with id bbabb62d3880ecda518c3b1346f775fed5497598b695590383f0b308a1a66cd6
Jul 26 15:50:58.632080 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:58.632051    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:50:58.769775 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:58.769718    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9wvxg" event={"ID":"7c34afa8-d2eb-4651-8464-4a56f7605447","Type":"ContainerStarted","Data":"bbabb62d3880ecda518c3b1346f775fed5497598b695590383f0b308a1a66cd6"}
Jul 26 15:50:58.833520 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:58.833466    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:50:59.242279 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:59.242244    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:50:59.342703 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:59.342671    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:50:59.442480 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:59.442426    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:50:59.863686 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:50:59.863650    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:00.631755 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:00.631700    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:00.781828 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:00.781794    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9wvxg" event={"ID":"7c34afa8-d2eb-4651-8464-4a56f7605447","Type":"ContainerStarted","Data":"2864044f3e53aef37452379908d4f8efd95703fc23808bed73ed6a7c217c2b86"}
Jul 26 15:51:00.782019 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:00.781870    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9wvxg" event={"ID":"7c34afa8-d2eb-4651-8464-4a56f7605447","Type":"ContainerStarted","Data":"43a7bfcc9b54cc7c4265246ec27022bdbbff7df77589feb4aff520f8e7349673"}
Jul 26 15:51:00.782019 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:00.781932    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-9wvxg"
Jul 26 15:51:00.797858 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:00.797805    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-9wvxg" podStartSLOduration=16.787517826 podStartE2EDuration="18.797788928s" totalImagesPullingTime="2.010271102s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:50:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:51:00.797120719 +0000 UTC m=+52.901346529" watchObservedRunningTime="2026-07-26 15:51:00.797788928 +0000 UTC m=+52.902014736"
Jul 26 15:51:00.839901 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:00.839870    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:01.234565 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:01.234530    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:01.438054 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:01.438026    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:01.546388 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:01.546302    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:01.842740 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:01.842660    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:02.435687 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:02.435656    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:03.033994 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:03.033957    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:03.243538 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:03.243505    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:03.640714 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:03.640684    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:03.838894 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:03.838868    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:04.248019 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:04.247971    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:04.347475 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:04.347432    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:04.444088 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:04.444028    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:04.544238 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:04.544154    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:04.602428 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:04.602397    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:04.958138 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:04.958058    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:05.059552 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:05.059507    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:05.172382 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:05.172341    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:05.364047 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:05.364004    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:05.972283 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:05.972255    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:06.306284 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:06.306252    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:06.407516 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:06.407472    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:06.986078 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:06.986025    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:07.716172 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:07.716142    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kwjfd"
Jul 26 15:51:08.936516 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:08.936482    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5885fdb759-lgfrd"]
Jul 26 15:51:08.939074 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:08.939054    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-lgfrd"
Jul 26 15:51:08.941301 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:08.941276    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 26 15:51:08.941401 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:08.941380    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-l9rgb\""
Jul 26 15:51:08.942171 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:08.942148    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 26 15:51:08.942265 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:08.942250    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 26 15:51:08.942308 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:08.942274    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 26 15:51:08.946631 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:08.946614    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 26 15:51:08.946738 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:08.946632    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-656db5f897-qkzdg"]
Jul 26 15:51:08.948472 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:08.948363    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-656db5f897-qkzdg"
Jul 26 15:51:08.949106 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:08.949085    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-lgfrd"]
Jul 26 15:51:08.952310 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:08.952292    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 26 15:51:08.952418 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:08.952395    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 26 15:51:08.952485 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:08.952461    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 26 15:51:08.952687 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:08.952670    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 26 15:51:08.952755 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:08.952698    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 26 15:51:08.952988 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:08.952968    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 26 15:51:08.955032 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:08.955013    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 26 15:51:08.967306 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:08.967282    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-656db5f897-qkzdg"]
Jul 26 15:51:09.015460 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:09.015433    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f135e47d-f8a0-413b-aed2-37d48bc1a2f7-tmp\") pod \"insights-operator-5885fdb759-lgfrd\" (UID: \"f135e47d-f8a0-413b-aed2-37d48bc1a2f7\") " pod="openshift-insights/insights-operator-5885fdb759-lgfrd"
Jul 26 15:51:09.015460 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:09.015462    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f135e47d-f8a0-413b-aed2-37d48bc1a2f7-service-ca-bundle\") pod \"insights-operator-5885fdb759-lgfrd\" (UID: \"f135e47d-f8a0-413b-aed2-37d48bc1a2f7\") " pod="openshift-insights/insights-operator-5885fdb759-lgfrd"
Jul 26 15:51:09.015665 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:09.015488    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d9a93ad-4f17-409c-902e-1bcdfb2da9e2-additional-service-ca\") pod \"cluster-proxy-proxy-agent-656db5f897-qkzdg\" (UID: \"2d9a93ad-4f17-409c-902e-1bcdfb2da9e2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-656db5f897-qkzdg"
Jul 26 15:51:09.015665 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:09.015519    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f135e47d-f8a0-413b-aed2-37d48bc1a2f7-snapshots\") pod \"insights-operator-5885fdb759-lgfrd\" (UID: \"f135e47d-f8a0-413b-aed2-37d48bc1a2f7\") " pod="openshift-insights/insights-operator-5885fdb759-lgfrd"
Jul 26 15:51:09.015665 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:09.015555    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f135e47d-f8a0-413b-aed2-37d48bc1a2f7-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-lgfrd\" (UID: \"f135e47d-f8a0-413b-aed2-37d48bc1a2f7\") " pod="openshift-insights/insights-operator-5885fdb759-lgfrd"
Jul 26 15:51:09.015665 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:09.015582    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2d9a93ad-4f17-409c-902e-1bcdfb2da9e2-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-656db5f897-qkzdg\" (UID: \"2d9a93ad-4f17-409c-902e-1bcdfb2da9e2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-656db5f897-qkzdg"
Jul 26 15:51:09.015665 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:09.015614    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/2d9a93ad-4f17-409c-902e-1bcdfb2da9e2-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-656db5f897-qkzdg\" (UID: \"2d9a93ad-4f17-409c-902e-1bcdfb2da9e2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-656db5f897-qkzdg"
Jul 26 15:51:09.015665 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:09.015633    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f135e47d-f8a0-413b-aed2-37d48bc1a2f7-serving-cert\") pod \"insights-operator-5885fdb759-lgfrd\" (UID: \"f135e47d-f8a0-413b-aed2-37d48bc1a2f7\") " pod="openshift-insights/insights-operator-5885fdb759-lgfrd"
Jul 26 15:51:09.015665 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:09.015647    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/2d9a93ad-4f17-409c-902e-1bcdfb2da9e2-hub\") pod \"cluster-proxy-proxy-agent-656db5f897-qkzdg\" (UID: \"2d9a93ad-4f17-409c-902e-1bcdfb2da9e2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-656db5f897-qkzdg"
Jul 26 15:51:09.015929 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:09.015691    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw8jv\" (UniqueName: \"kubernetes.io/projected/f135e47d-f8a0-413b-aed2-37d48bc1a2f7-kube-api-access-qw8jv\") pod \"insights-operator-5885fdb759-lgfrd\" (UID: \"f135e47d-f8a0-413b-aed2-37d48bc1a2f7\") " pod="openshift-insights/insights-operator-5885fdb759-lgfrd"
Jul 26 15:51:09.015929 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:09.015715    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wncmw\" (UniqueName: \"kubernetes.io/projected/2d9a93ad-4f17-409c-902e-1bcdfb2da9e2-kube-api-access-wncmw\") pod \"cluster-proxy-proxy-agent-656db5f897-qkzdg\" (UID: \"2d9a93ad-4f17-409c-902e-1bcdfb2da9e2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-656db5f897-qkzdg"
Jul 26 15:51:09.015929 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:09.015738    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/2d9a93ad-4f17-409c-902e-1bcdfb2da9e2-ca\") pod \"cluster-proxy-proxy-agent-656db5f897-qkzdg\" (UID: \"2d9a93ad-4f17-409c-902e-1bcdfb2da9e2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-656db5f897-qkzdg"
Jul 26 15:51:09.095464 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:09.095428    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:09.116557 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:09.116522    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d9a93ad-4f17-409c-902e-1bcdfb2da9e2-additional-service-ca\") pod \"cluster-proxy-proxy-agent-656db5f897-qkzdg\" (UID: \"2d9a93ad-4f17-409c-902e-1bcdfb2da9e2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-656db5f897-qkzdg"
Jul 26 15:51:09.116723 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:09.116568    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f135e47d-f8a0-413b-aed2-37d48bc1a2f7-snapshots\") pod \"insights-operator-5885fdb759-lgfrd\" (UID: \"f135e47d-f8a0-413b-aed2-37d48bc1a2f7\") " pod="openshift-insights/insights-operator-5885fdb759-lgfrd"
Jul 26 15:51:09.116723 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:09.116588    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f135e47d-f8a0-413b-aed2-37d48bc1a2f7-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-lgfrd\" (UID: \"f135e47d-f8a0-413b-aed2-37d48bc1a2f7\") " pod="openshift-insights/insights-operator-5885fdb759-lgfrd"
Jul 26 15:51:09.116723 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:09.116605    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2d9a93ad-4f17-409c-902e-1bcdfb2da9e2-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-656db5f897-qkzdg\" (UID: \"2d9a93ad-4f17-409c-902e-1bcdfb2da9e2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-656db5f897-qkzdg"
Jul 26 15:51:09.116723 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:09.116625    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/2d9a93ad-4f17-409c-902e-1bcdfb2da9e2-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-656db5f897-qkzdg\" (UID: \"2d9a93ad-4f17-409c-902e-1bcdfb2da9e2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-656db5f897-qkzdg"
Jul 26 15:51:09.116723 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:09.116672    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f135e47d-f8a0-413b-aed2-37d48bc1a2f7-serving-cert\") pod \"insights-operator-5885fdb759-lgfrd\" (UID: \"f135e47d-f8a0-413b-aed2-37d48bc1a2f7\") " pod="openshift-insights/insights-operator-5885fdb759-lgfrd"
Jul 26 15:51:09.116723 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:09.116688    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/2d9a93ad-4f17-409c-902e-1bcdfb2da9e2-hub\") pod \"cluster-proxy-proxy-agent-656db5f897-qkzdg\" (UID: \"2d9a93ad-4f17-409c-902e-1bcdfb2da9e2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-656db5f897-qkzdg"
Jul 26 15:51:09.116723 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:09.116716    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw8jv\" (UniqueName: \"kubernetes.io/projected/f135e47d-f8a0-413b-aed2-37d48bc1a2f7-kube-api-access-qw8jv\") pod \"insights-operator-5885fdb759-lgfrd\" (UID: \"f135e47d-f8a0-413b-aed2-37d48bc1a2f7\") " pod="openshift-insights/insights-operator-5885fdb759-lgfrd"
Jul 26 15:51:09.117127 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:09.116736    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wncmw\" (UniqueName: \"kubernetes.io/projected/2d9a93ad-4f17-409c-902e-1bcdfb2da9e2-kube-api-access-wncmw\") pod \"cluster-proxy-proxy-agent-656db5f897-qkzdg\" (UID: \"2d9a93ad-4f17-409c-902e-1bcdfb2da9e2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-656db5f897-qkzdg"
Jul 26 15:51:09.117127 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:09.116754    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/2d9a93ad-4f17-409c-902e-1bcdfb2da9e2-ca\") pod \"cluster-proxy-proxy-agent-656db5f897-qkzdg\" (UID: \"2d9a93ad-4f17-409c-902e-1bcdfb2da9e2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-656db5f897-qkzdg"
Jul 26 15:51:09.117127 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:09.116829    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f135e47d-f8a0-413b-aed2-37d48bc1a2f7-tmp\") pod \"insights-operator-5885fdb759-lgfrd\" (UID: \"f135e47d-f8a0-413b-aed2-37d48bc1a2f7\") " pod="openshift-insights/insights-operator-5885fdb759-lgfrd"
Jul 26 15:51:09.117127 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:09.116852    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f135e47d-f8a0-413b-aed2-37d48bc1a2f7-service-ca-bundle\") pod \"insights-operator-5885fdb759-lgfrd\" (UID: \"f135e47d-f8a0-413b-aed2-37d48bc1a2f7\") " pod="openshift-insights/insights-operator-5885fdb759-lgfrd"
Jul 26 15:51:09.117387 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:09.117338    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f135e47d-f8a0-413b-aed2-37d48bc1a2f7-tmp\") pod \"insights-operator-5885fdb759-lgfrd\" (UID: \"f135e47d-f8a0-413b-aed2-37d48bc1a2f7\") " pod="openshift-insights/insights-operator-5885fdb759-lgfrd"
Jul 26 15:51:09.117447 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:09.117396    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f135e47d-f8a0-413b-aed2-37d48bc1a2f7-snapshots\") pod \"insights-operator-5885fdb759-lgfrd\" (UID: \"f135e47d-f8a0-413b-aed2-37d48bc1a2f7\") " pod="openshift-insights/insights-operator-5885fdb759-lgfrd"
Jul 26 15:51:09.117719 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:09.117694    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f135e47d-f8a0-413b-aed2-37d48bc1a2f7-service-ca-bundle\") pod \"insights-operator-5885fdb759-lgfrd\" (UID: \"f135e47d-f8a0-413b-aed2-37d48bc1a2f7\") " pod="openshift-insights/insights-operator-5885fdb759-lgfrd"
Jul 26 15:51:09.117947 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:09.117918    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f135e47d-f8a0-413b-aed2-37d48bc1a2f7-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-lgfrd\" (UID: \"f135e47d-f8a0-413b-aed2-37d48bc1a2f7\") " pod="openshift-insights/insights-operator-5885fdb759-lgfrd"
Jul 26 15:51:09.118096 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:09.118076    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d9a93ad-4f17-409c-902e-1bcdfb2da9e2-additional-service-ca\") pod \"cluster-proxy-proxy-agent-656db5f897-qkzdg\" (UID: \"2d9a93ad-4f17-409c-902e-1bcdfb2da9e2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-656db5f897-qkzdg"
Jul 26 15:51:09.119515 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:09.119493    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f135e47d-f8a0-413b-aed2-37d48bc1a2f7-serving-cert\") pod \"insights-operator-5885fdb759-lgfrd\" (UID: \"f135e47d-f8a0-413b-aed2-37d48bc1a2f7\") " pod="openshift-insights/insights-operator-5885fdb759-lgfrd"
Jul 26 15:51:09.119795 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:09.119756    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2d9a93ad-4f17-409c-902e-1bcdfb2da9e2-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-656db5f897-qkzdg\" (UID: \"2d9a93ad-4f17-409c-902e-1bcdfb2da9e2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-656db5f897-qkzdg"
Jul 26 15:51:09.119904 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:09.119880    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/2d9a93ad-4f17-409c-902e-1bcdfb2da9e2-hub\") pod \"cluster-proxy-proxy-agent-656db5f897-qkzdg\" (UID: \"2d9a93ad-4f17-409c-902e-1bcdfb2da9e2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-656db5f897-qkzdg"
Jul 26 15:51:09.120012 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:09.119991    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/2d9a93ad-4f17-409c-902e-1bcdfb2da9e2-ca\") pod \"cluster-proxy-proxy-agent-656db5f897-qkzdg\" (UID: \"2d9a93ad-4f17-409c-902e-1bcdfb2da9e2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-656db5f897-qkzdg"
Jul 26 15:51:09.120073 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:09.120040    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/2d9a93ad-4f17-409c-902e-1bcdfb2da9e2-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-656db5f897-qkzdg\" (UID: \"2d9a93ad-4f17-409c-902e-1bcdfb2da9e2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-656db5f897-qkzdg"
Jul 26 15:51:09.125277 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:09.125249    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wncmw\" (UniqueName: \"kubernetes.io/projected/2d9a93ad-4f17-409c-902e-1bcdfb2da9e2-kube-api-access-wncmw\") pod \"cluster-proxy-proxy-agent-656db5f897-qkzdg\" (UID: \"2d9a93ad-4f17-409c-902e-1bcdfb2da9e2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-656db5f897-qkzdg"
Jul 26 15:51:09.125916 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:09.125897    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw8jv\" (UniqueName: \"kubernetes.io/projected/f135e47d-f8a0-413b-aed2-37d48bc1a2f7-kube-api-access-qw8jv\") pod \"insights-operator-5885fdb759-lgfrd\" (UID: \"f135e47d-f8a0-413b-aed2-37d48bc1a2f7\") " pod="openshift-insights/insights-operator-5885fdb759-lgfrd"
Jul 26 15:51:09.180470 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:09.180442    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:09.249026 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:09.248998    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-lgfrd"
Jul 26 15:51:09.254177 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:09.254142    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:09.268135 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:09.268110    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-656db5f897-qkzdg"
Jul 26 15:51:09.353306 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:09.353264    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:09.383043 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:09.383009    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-lgfrd"]
Jul 26 15:51:09.386642 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:51:09.386612    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf135e47d_f8a0_413b_aed2_37d48bc1a2f7.slice/crio-09844ad2494036e6e1c95484e4312c5d143999097148bcdd86a1ee3a26c1b8cf WatchSource:0}: Error finding container 09844ad2494036e6e1c95484e4312c5d143999097148bcdd86a1ee3a26c1b8cf: Status 404 returned error can't find the container with id 09844ad2494036e6e1c95484e4312c5d143999097148bcdd86a1ee3a26c1b8cf
Jul 26 15:51:09.401849 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:09.401825    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-656db5f897-qkzdg"]
Jul 26 15:51:09.404714 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:51:09.404691    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2d9a93ad_4f17_409c_902e_1bcdfb2da9e2.slice/crio-c7e761f6029db5bd0db96c904e75724bf3a6935eebba8959f0ce5b991c1a6535 WatchSource:0}: Error finding container c7e761f6029db5bd0db96c904e75724bf3a6935eebba8959f0ce5b991c1a6535: Status 404 returned error can't find the container with id c7e761f6029db5bd0db96c904e75724bf3a6935eebba8959f0ce5b991c1a6535
Jul 26 15:51:09.810713 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:09.810671    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-lgfrd" event={"ID":"f135e47d-f8a0-413b-aed2-37d48bc1a2f7","Type":"ContainerStarted","Data":"09844ad2494036e6e1c95484e4312c5d143999097148bcdd86a1ee3a26c1b8cf"}
Jul 26 15:51:09.811722 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:09.811699    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-656db5f897-qkzdg" event={"ID":"2d9a93ad-4f17-409c-902e-1bcdfb2da9e2","Type":"ContainerStarted","Data":"c7e761f6029db5bd0db96c904e75724bf3a6935eebba8959f0ce5b991c1a6535"}
Jul 26 15:51:10.788235 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:10.788207    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-9wvxg"
Jul 26 15:51:10.919037 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:10.918986    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:11.020683 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:11.020641    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:11.730564 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:11.730524    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:11.831196 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:11.831156    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:12.797989 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:12.797951    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:12.859358 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:12.859322    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:13.149802 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:13.149704    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e62cc557-6267-4993-84a1-a888ced92345-original-pull-secret\") pod \"global-pull-secret-syncer-cxfp4\" (UID: \"e62cc557-6267-4993-84a1-a888ced92345\") " pod="kube-system/global-pull-secret-syncer-cxfp4"
Jul 26 15:51:13.149802 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:13.149772    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca110f78-ba25-4be3-9bed-60e5ef15e2c5-metrics-certs\") pod \"network-metrics-daemon-8gzcl\" (UID: \"ca110f78-ba25-4be3-9bed-60e5ef15e2c5\") " pod="openshift-multus/network-metrics-daemon-8gzcl"
Jul 26 15:51:13.152254 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:13.152230    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 26 15:51:13.152359 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:13.152262    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 26 15:51:13.162555 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:13.162529    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e62cc557-6267-4993-84a1-a888ced92345-original-pull-secret\") pod \"global-pull-secret-syncer-cxfp4\" (UID: \"e62cc557-6267-4993-84a1-a888ced92345\") " pod="kube-system/global-pull-secret-syncer-cxfp4"
Jul 26 15:51:13.162555 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:13.162551    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca110f78-ba25-4be3-9bed-60e5ef15e2c5-metrics-certs\") pod \"network-metrics-daemon-8gzcl\" (UID: \"ca110f78-ba25-4be3-9bed-60e5ef15e2c5\") " pod="openshift-multus/network-metrics-daemon-8gzcl"
Jul 26 15:51:13.250347 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:13.250304    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmxk6\" (UniqueName: \"kubernetes.io/projected/0c90caab-1908-4071-a259-e0343960cbe2-kube-api-access-bmxk6\") pod \"network-check-target-7g75t\" (UID: \"0c90caab-1908-4071-a259-e0343960cbe2\") " pod="openshift-network-diagnostics/network-check-target-7g75t"
Jul 26 15:51:13.252972 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:13.252949    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 26 15:51:13.263049 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:13.263025    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 26 15:51:13.273496 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:13.273471    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmxk6\" (UniqueName: \"kubernetes.io/projected/0c90caab-1908-4071-a259-e0343960cbe2-kube-api-access-bmxk6\") pod \"network-check-target-7g75t\" (UID: \"0c90caab-1908-4071-a259-e0343960cbe2\") " pod="openshift-network-diagnostics/network-check-target-7g75t"
Jul 26 15:51:13.280723 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:13.280697    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-c5h9x\""
Jul 26 15:51:13.286726 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:13.286704    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-r7gjf\""
Jul 26 15:51:13.288228 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:13.288206    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7g75t"
Jul 26 15:51:13.290287 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:13.290268    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cxfp4"
Jul 26 15:51:13.295772 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:13.295740    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8gzcl"
Jul 26 15:51:13.606092 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:13.605722    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7g75t"]
Jul 26 15:51:13.620028 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:13.619949    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8gzcl"]
Jul 26 15:51:13.625738 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:51:13.625488    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca110f78_ba25_4be3_9bed_60e5ef15e2c5.slice/crio-f95371be3ad20e07ee221473590545c8adaaf66aaf82aa78eba62ee423546bef WatchSource:0}: Error finding container f95371be3ad20e07ee221473590545c8adaaf66aaf82aa78eba62ee423546bef: Status 404 returned error can't find the container with id f95371be3ad20e07ee221473590545c8adaaf66aaf82aa78eba62ee423546bef
Jul 26 15:51:13.642612 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:13.642579    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cxfp4"]
Jul 26 15:51:13.646666 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:51:13.646611    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode62cc557_6267_4993_84a1_a888ced92345.slice/crio-b8fd6570418e68ab80f941f4facde7bf41fe781948b6f336cbce2c28ddcf6fd0 WatchSource:0}: Error finding container b8fd6570418e68ab80f941f4facde7bf41fe781948b6f336cbce2c28ddcf6fd0: Status 404 returned error can't find the container with id b8fd6570418e68ab80f941f4facde7bf41fe781948b6f336cbce2c28ddcf6fd0
Jul 26 15:51:13.830733 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:13.830695    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7g75t" event={"ID":"0c90caab-1908-4071-a259-e0343960cbe2","Type":"ContainerStarted","Data":"77782e5285109b0b9d96c09558c28be4315e2798902d8e62e10397c58fa9c1ed"}
Jul 26 15:51:13.831827 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:13.831801    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cxfp4" event={"ID":"e62cc557-6267-4993-84a1-a888ced92345","Type":"ContainerStarted","Data":"b8fd6570418e68ab80f941f4facde7bf41fe781948b6f336cbce2c28ddcf6fd0"}
Jul 26 15:51:13.833359 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:13.833335    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-lgfrd" event={"ID":"f135e47d-f8a0-413b-aed2-37d48bc1a2f7","Type":"ContainerStarted","Data":"8cd5df2f6d66cfbeabbf09db52669c21e8b120c2401497b12ebf64f4cc4db12e"}
Jul 26 15:51:13.834653 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:13.834624    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8gzcl" event={"ID":"ca110f78-ba25-4be3-9bed-60e5ef15e2c5","Type":"ContainerStarted","Data":"f95371be3ad20e07ee221473590545c8adaaf66aaf82aa78eba62ee423546bef"}
Jul 26 15:51:13.836546 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:13.836523    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-656db5f897-qkzdg" event={"ID":"2d9a93ad-4f17-409c-902e-1bcdfb2da9e2","Type":"ContainerStarted","Data":"9f1eaddd51950d9cf6f34083b8a64b1d51eac8f8cf219e0abc56e8acff4f23dd"}
Jul 26 15:51:13.836663 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:13.836558    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-656db5f897-qkzdg" event={"ID":"2d9a93ad-4f17-409c-902e-1bcdfb2da9e2","Type":"ContainerStarted","Data":"3c0d62193a2e33f887b2b3c577d938cca2a2fefd91ab3ba419b1e6418d8c56d4"}
Jul 26 15:51:13.836663 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:13.836572    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-656db5f897-qkzdg" event={"ID":"2d9a93ad-4f17-409c-902e-1bcdfb2da9e2","Type":"ContainerStarted","Data":"34669e2935c7c46d1513ae3e98b6bc83a309f1f1e238d489fb268a4b4ed2565b"}
Jul 26 15:51:13.850711 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:13.850666    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5885fdb759-lgfrd" podStartSLOduration=1.791988066 podStartE2EDuration="5.850649586s" totalImagesPullingTime="4.05866152s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:51:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:51:13.849933007 +0000 UTC m=+65.954158827" watchObservedRunningTime="2026-07-26 15:51:13.850649586 +0000 UTC m=+65.954875395"
Jul 26 15:51:13.866053 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:13.865987    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-656db5f897-qkzdg" podStartSLOduration=1.805998898 podStartE2EDuration="5.86596546s" totalImagesPullingTime="4.059966562s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:51:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:51:13.865663588 +0000 UTC m=+65.969889398" watchObservedRunningTime="2026-07-26 15:51:13.86596546 +0000 UTC m=+65.970191269"
Jul 26 15:51:14.489377 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:14.489062    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-6shks"]
Jul 26 15:51:14.507897 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:14.507860    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6shks"]
Jul 26 15:51:14.508081 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:14.508010    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6shks"
Jul 26 15:51:14.514295 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:14.512907    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 26 15:51:14.514295 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:14.513198    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 26 15:51:14.514295 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:14.513416    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-lf682\""
Jul 26 15:51:14.530331 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:14.530287    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:14.561445 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:14.561409    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/212d7a6b-fb00-48bf-88a1-c5d617abe2cb-crio-socket\") pod \"insights-runtime-extractor-6shks\" (UID: \"212d7a6b-fb00-48bf-88a1-c5d617abe2cb\") " pod="openshift-insights/insights-runtime-extractor-6shks"
Jul 26 15:51:14.561445 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:14.561448    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/212d7a6b-fb00-48bf-88a1-c5d617abe2cb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6shks\" (UID: \"212d7a6b-fb00-48bf-88a1-c5d617abe2cb\") " pod="openshift-insights/insights-runtime-extractor-6shks"
Jul 26 15:51:14.561713 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:14.561473    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/212d7a6b-fb00-48bf-88a1-c5d617abe2cb-data-volume\") pod \"insights-runtime-extractor-6shks\" (UID: \"212d7a6b-fb00-48bf-88a1-c5d617abe2cb\") " pod="openshift-insights/insights-runtime-extractor-6shks"
Jul 26 15:51:14.561713 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:14.561492    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/212d7a6b-fb00-48bf-88a1-c5d617abe2cb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6shks\" (UID: \"212d7a6b-fb00-48bf-88a1-c5d617abe2cb\") " pod="openshift-insights/insights-runtime-extractor-6shks"
Jul 26 15:51:14.561713 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:14.561516    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-669ds\" (UniqueName: \"kubernetes.io/projected/212d7a6b-fb00-48bf-88a1-c5d617abe2cb-kube-api-access-669ds\") pod \"insights-runtime-extractor-6shks\" (UID: \"212d7a6b-fb00-48bf-88a1-c5d617abe2cb\") " pod="openshift-insights/insights-runtime-extractor-6shks"
Jul 26 15:51:14.662297 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:14.662263    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-669ds\" (UniqueName: \"kubernetes.io/projected/212d7a6b-fb00-48bf-88a1-c5d617abe2cb-kube-api-access-669ds\") pod \"insights-runtime-extractor-6shks\" (UID: \"212d7a6b-fb00-48bf-88a1-c5d617abe2cb\") " pod="openshift-insights/insights-runtime-extractor-6shks"
Jul 26 15:51:14.662500 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:14.662336    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/212d7a6b-fb00-48bf-88a1-c5d617abe2cb-crio-socket\") pod \"insights-runtime-extractor-6shks\" (UID: \"212d7a6b-fb00-48bf-88a1-c5d617abe2cb\") " pod="openshift-insights/insights-runtime-extractor-6shks"
Jul 26 15:51:14.662500 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:14.662366    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/212d7a6b-fb00-48bf-88a1-c5d617abe2cb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6shks\" (UID: \"212d7a6b-fb00-48bf-88a1-c5d617abe2cb\") " pod="openshift-insights/insights-runtime-extractor-6shks"
Jul 26 15:51:14.662500 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:14.662406    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/212d7a6b-fb00-48bf-88a1-c5d617abe2cb-data-volume\") pod \"insights-runtime-extractor-6shks\" (UID: \"212d7a6b-fb00-48bf-88a1-c5d617abe2cb\") " pod="openshift-insights/insights-runtime-extractor-6shks"
Jul 26 15:51:14.662500 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:14.662436    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/212d7a6b-fb00-48bf-88a1-c5d617abe2cb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6shks\" (UID: \"212d7a6b-fb00-48bf-88a1-c5d617abe2cb\") " pod="openshift-insights/insights-runtime-extractor-6shks"
Jul 26 15:51:14.662706 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:51:14.662532    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 26 15:51:14.662706 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:51:14.662608    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/212d7a6b-fb00-48bf-88a1-c5d617abe2cb-kube-rbac-proxy-cm podName:212d7a6b-fb00-48bf-88a1-c5d617abe2cb nodeName:}" failed. No retries permitted until 2026-07-26 15:51:15.162586404 +0000 UTC m=+67.266812210 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/212d7a6b-fb00-48bf-88a1-c5d617abe2cb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6shks" (UID: "212d7a6b-fb00-48bf-88a1-c5d617abe2cb") : configmap "kube-rbac-proxy" not found
Jul 26 15:51:14.662706 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:51:14.662695    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 26 15:51:14.662907 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:51:14.662751    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/212d7a6b-fb00-48bf-88a1-c5d617abe2cb-insights-runtime-extractor-tls podName:212d7a6b-fb00-48bf-88a1-c5d617abe2cb nodeName:}" failed. No retries permitted until 2026-07-26 15:51:15.162732969 +0000 UTC m=+67.266958763 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/212d7a6b-fb00-48bf-88a1-c5d617abe2cb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6shks" (UID: "212d7a6b-fb00-48bf-88a1-c5d617abe2cb") : secret "insights-runtime-extractor-tls" not found
Jul 26 15:51:14.662907 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:14.662833    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/212d7a6b-fb00-48bf-88a1-c5d617abe2cb-crio-socket\") pod \"insights-runtime-extractor-6shks\" (UID: \"212d7a6b-fb00-48bf-88a1-c5d617abe2cb\") " pod="openshift-insights/insights-runtime-extractor-6shks"
Jul 26 15:51:14.663189 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:14.663163    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/212d7a6b-fb00-48bf-88a1-c5d617abe2cb-data-volume\") pod \"insights-runtime-extractor-6shks\" (UID: \"212d7a6b-fb00-48bf-88a1-c5d617abe2cb\") " pod="openshift-insights/insights-runtime-extractor-6shks"
Jul 26 15:51:14.673828 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:14.673741    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-669ds\" (UniqueName: \"kubernetes.io/projected/212d7a6b-fb00-48bf-88a1-c5d617abe2cb-kube-api-access-669ds\") pod \"insights-runtime-extractor-6shks\" (UID: \"212d7a6b-fb00-48bf-88a1-c5d617abe2cb\") " pod="openshift-insights/insights-runtime-extractor-6shks"
Jul 26 15:51:15.092528 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:15.092490    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9wvxg_7c34afa8-d2eb-4651-8464-4a56f7605447/dns/0.log"
Jul 26 15:51:15.168601 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:15.168437    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/212d7a6b-fb00-48bf-88a1-c5d617abe2cb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6shks\" (UID: \"212d7a6b-fb00-48bf-88a1-c5d617abe2cb\") " pod="openshift-insights/insights-runtime-extractor-6shks"
Jul 26 15:51:15.168601 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:15.168518    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/212d7a6b-fb00-48bf-88a1-c5d617abe2cb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6shks\" (UID: \"212d7a6b-fb00-48bf-88a1-c5d617abe2cb\") " pod="openshift-insights/insights-runtime-extractor-6shks"
Jul 26 15:51:15.168601 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:51:15.168582    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 26 15:51:15.168879 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:51:15.168648    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/212d7a6b-fb00-48bf-88a1-c5d617abe2cb-insights-runtime-extractor-tls podName:212d7a6b-fb00-48bf-88a1-c5d617abe2cb nodeName:}" failed. No retries permitted until 2026-07-26 15:51:16.168628893 +0000 UTC m=+68.272854695 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/212d7a6b-fb00-48bf-88a1-c5d617abe2cb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6shks" (UID: "212d7a6b-fb00-48bf-88a1-c5d617abe2cb") : secret "insights-runtime-extractor-tls" not found
Jul 26 15:51:15.168879 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:51:15.168657    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 26 15:51:15.168879 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:51:15.168724    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/212d7a6b-fb00-48bf-88a1-c5d617abe2cb-kube-rbac-proxy-cm podName:212d7a6b-fb00-48bf-88a1-c5d617abe2cb nodeName:}" failed. No retries permitted until 2026-07-26 15:51:16.168706036 +0000 UTC m=+68.272931827 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/212d7a6b-fb00-48bf-88a1-c5d617abe2cb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6shks" (UID: "212d7a6b-fb00-48bf-88a1-c5d617abe2cb") : configmap "kube-rbac-proxy" not found
Jul 26 15:51:15.205862 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:15.205745    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:15.292147 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:15.292091    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9wvxg_7c34afa8-d2eb-4651-8464-4a56f7605447/kube-rbac-proxy/0.log"
Jul 26 15:51:15.848987 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:15.848950    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8gzcl" event={"ID":"ca110f78-ba25-4be3-9bed-60e5ef15e2c5","Type":"ContainerStarted","Data":"02ef546f8bc1afa0a2e8234a4e24646c36a1574ef220c80a21c529c66a585517"}
Jul 26 15:51:15.849176 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:15.849047    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8gzcl" event={"ID":"ca110f78-ba25-4be3-9bed-60e5ef15e2c5","Type":"ContainerStarted","Data":"31d1f5061d874e9be68a8911956f7703b5bc1dbae70a07ace9529c6ced23c937"}
Jul 26 15:51:15.865398 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:15.865348    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-8gzcl" podStartSLOduration=65.36191073 podStartE2EDuration="1m6.865329178s" totalImagesPullingTime="1.503418448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:50:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:51:15.86430551 +0000 UTC m=+67.968531321" watchObservedRunningTime="2026-07-26 15:51:15.865329178 +0000 UTC m=+67.969554988"
Jul 26 15:51:15.956864 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:15.956739    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:16.013485 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:16.013415    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:16.177270 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:16.177132    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/212d7a6b-fb00-48bf-88a1-c5d617abe2cb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6shks\" (UID: \"212d7a6b-fb00-48bf-88a1-c5d617abe2cb\") " pod="openshift-insights/insights-runtime-extractor-6shks"
Jul 26 15:51:16.177270 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:16.177197    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/212d7a6b-fb00-48bf-88a1-c5d617abe2cb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6shks\" (UID: \"212d7a6b-fb00-48bf-88a1-c5d617abe2cb\") " pod="openshift-insights/insights-runtime-extractor-6shks"
Jul 26 15:51:16.177822 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:51:16.177307    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 26 15:51:16.177822 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:51:16.177383    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 26 15:51:16.177822 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:51:16.177390    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/212d7a6b-fb00-48bf-88a1-c5d617abe2cb-kube-rbac-proxy-cm podName:212d7a6b-fb00-48bf-88a1-c5d617abe2cb nodeName:}" failed. No retries permitted until 2026-07-26 15:51:18.177366638 +0000 UTC m=+70.281592466 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/212d7a6b-fb00-48bf-88a1-c5d617abe2cb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6shks" (UID: "212d7a6b-fb00-48bf-88a1-c5d617abe2cb") : configmap "kube-rbac-proxy" not found
Jul 26 15:51:16.177822 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:51:16.177456    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/212d7a6b-fb00-48bf-88a1-c5d617abe2cb-insights-runtime-extractor-tls podName:212d7a6b-fb00-48bf-88a1-c5d617abe2cb nodeName:}" failed. No retries permitted until 2026-07-26 15:51:18.177439451 +0000 UTC m=+70.281665241 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/212d7a6b-fb00-48bf-88a1-c5d617abe2cb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6shks" (UID: "212d7a6b-fb00-48bf-88a1-c5d617abe2cb") : secret "insights-runtime-extractor-tls" not found
Jul 26 15:51:16.491947 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:16.491924    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gnkxb_aba3d036-e332-4b06-8379-7deb395345ba/dns-node-resolver/0.log"
Jul 26 15:51:17.292476 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:17.292448    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-p8kcf_e1f2fc7a-9737-422c-9c17-c8821dc48396/node-ca/0.log"
Jul 26 15:51:17.888786 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:17.888734    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:17.947751 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:17.947703    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:18.010477 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:18.010442    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:18.194645 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:18.194525    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/212d7a6b-fb00-48bf-88a1-c5d617abe2cb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6shks\" (UID: \"212d7a6b-fb00-48bf-88a1-c5d617abe2cb\") " pod="openshift-insights/insights-runtime-extractor-6shks"
Jul 26 15:51:18.194645 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:18.194631    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/212d7a6b-fb00-48bf-88a1-c5d617abe2cb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6shks\" (UID: \"212d7a6b-fb00-48bf-88a1-c5d617abe2cb\") " pod="openshift-insights/insights-runtime-extractor-6shks"
Jul 26 15:51:18.194891 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:51:18.194706    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 26 15:51:18.194891 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:51:18.194716    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 26 15:51:18.194891 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:51:18.194789    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/212d7a6b-fb00-48bf-88a1-c5d617abe2cb-kube-rbac-proxy-cm podName:212d7a6b-fb00-48bf-88a1-c5d617abe2cb nodeName:}" failed. No retries permitted until 2026-07-26 15:51:22.194771521 +0000 UTC m=+74.298997325 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/212d7a6b-fb00-48bf-88a1-c5d617abe2cb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6shks" (UID: "212d7a6b-fb00-48bf-88a1-c5d617abe2cb") : configmap "kube-rbac-proxy" not found
Jul 26 15:51:18.194891 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:51:18.194809    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/212d7a6b-fb00-48bf-88a1-c5d617abe2cb-insights-runtime-extractor-tls podName:212d7a6b-fb00-48bf-88a1-c5d617abe2cb nodeName:}" failed. No retries permitted until 2026-07-26 15:51:22.194799456 +0000 UTC m=+74.299025242 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/212d7a6b-fb00-48bf-88a1-c5d617abe2cb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6shks" (UID: "212d7a6b-fb00-48bf-88a1-c5d617abe2cb") : secret "insights-runtime-extractor-tls" not found
Jul 26 15:51:18.254985 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:18.254955    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:18.292368 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:18.292327    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9mkmv_214a33cc-fbe9-4dd9-8f42-14e737ac0341/serve-healthcheck-canary/0.log"
Jul 26 15:51:18.354416 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:18.354380    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:18.860975 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:18.860928    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7g75t" event={"ID":"0c90caab-1908-4071-a259-e0343960cbe2","Type":"ContainerStarted","Data":"aab317988f4822f2b1b04ce46cafe60a1d2a199c747c3446bb1981c1996cf432"}
Jul 26 15:51:18.861154 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:18.861032    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7g75t"
Jul 26 15:51:18.862389 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:18.862367    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cxfp4" event={"ID":"e62cc557-6267-4993-84a1-a888ced92345","Type":"ContainerStarted","Data":"fd78533fd00c32011b4f4ccdc1b2358648c86aa812251d7c22c43d2fa341917a"}
Jul 26 15:51:18.870836 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:18.870812    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:18.882189 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:18.882146    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-7g75t" podStartSLOduration=65.26959637 podStartE2EDuration="1m9.882133755s" totalImagesPullingTime="4.612537385s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:50:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:51:18.88204933 +0000 UTC m=+70.986275142" watchObservedRunningTime="2026-07-26 15:51:18.882133755 +0000 UTC m=+70.986359565"
Jul 26 15:51:18.905544 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:18.905502    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-cxfp4" podStartSLOduration=65.323183917 podStartE2EDuration="1m9.90548725s" totalImagesPullingTime="4.582303333s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 15:50:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:51:18.905202884 +0000 UTC m=+71.009428694" watchObservedRunningTime="2026-07-26 15:51:18.90548725 +0000 UTC m=+71.009713058"
Jul 26 15:51:18.945133 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:18.945105    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:19.037092 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:19.037056    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:19.190805 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:19.190690    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:19.290946 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:19.290903    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:19.653731 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:19.653701    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:19.753877 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:19.753845    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:19.911623 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:19.911545    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:19.997669 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:19.997641    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:20.460088 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:20.460053    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:20.561101 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:20.561060    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:21.800115 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:21.800060    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:21.859017 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:21.858979    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:22.065467 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:22.065392    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:22.132925 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:22.132895    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:22.221480 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:22.221447    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/212d7a6b-fb00-48bf-88a1-c5d617abe2cb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6shks\" (UID: \"212d7a6b-fb00-48bf-88a1-c5d617abe2cb\") " pod="openshift-insights/insights-runtime-extractor-6shks"
Jul 26 15:51:22.221635 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:22.221502    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/212d7a6b-fb00-48bf-88a1-c5d617abe2cb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6shks\" (UID: \"212d7a6b-fb00-48bf-88a1-c5d617abe2cb\") " pod="openshift-insights/insights-runtime-extractor-6shks"
Jul 26 15:51:22.221635 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:51:22.221597    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 26 15:51:22.221635 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:51:22.221613    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 26 15:51:22.221741 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:51:22.221666    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/212d7a6b-fb00-48bf-88a1-c5d617abe2cb-kube-rbac-proxy-cm podName:212d7a6b-fb00-48bf-88a1-c5d617abe2cb nodeName:}" failed. No retries permitted until 2026-07-26 15:51:30.221651004 +0000 UTC m=+82.325876792 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/212d7a6b-fb00-48bf-88a1-c5d617abe2cb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6shks" (UID: "212d7a6b-fb00-48bf-88a1-c5d617abe2cb") : configmap "kube-rbac-proxy" not found
Jul 26 15:51:22.221741 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:51:22.221681    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/212d7a6b-fb00-48bf-88a1-c5d617abe2cb-insights-runtime-extractor-tls podName:212d7a6b-fb00-48bf-88a1-c5d617abe2cb nodeName:}" failed. No retries permitted until 2026-07-26 15:51:30.221674908 +0000 UTC m=+82.325900695 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/212d7a6b-fb00-48bf-88a1-c5d617abe2cb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6shks" (UID: "212d7a6b-fb00-48bf-88a1-c5d617abe2cb") : secret "insights-runtime-extractor-tls" not found
Jul 26 15:51:22.481977 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:22.481948    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:25.797990 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:25.797956    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:25.988929 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:25.988890    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:26.089583 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:26.089501    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:29.538519 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:29.538487    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:29.642715 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:29.642683    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:30.281139 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:30.281012    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/212d7a6b-fb00-48bf-88a1-c5d617abe2cb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6shks\" (UID: \"212d7a6b-fb00-48bf-88a1-c5d617abe2cb\") " pod="openshift-insights/insights-runtime-extractor-6shks"
Jul 26 15:51:30.281139 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:30.281088    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/212d7a6b-fb00-48bf-88a1-c5d617abe2cb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6shks\" (UID: \"212d7a6b-fb00-48bf-88a1-c5d617abe2cb\") " pod="openshift-insights/insights-runtime-extractor-6shks"
Jul 26 15:51:30.281615 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:30.281594    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/212d7a6b-fb00-48bf-88a1-c5d617abe2cb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6shks\" (UID: \"212d7a6b-fb00-48bf-88a1-c5d617abe2cb\") " pod="openshift-insights/insights-runtime-extractor-6shks"
Jul 26 15:51:30.283329 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:30.283308    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/212d7a6b-fb00-48bf-88a1-c5d617abe2cb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6shks\" (UID: \"212d7a6b-fb00-48bf-88a1-c5d617abe2cb\") " pod="openshift-insights/insights-runtime-extractor-6shks"
Jul 26 15:51:30.397131 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:30.397095    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:30.424097 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:30.424067    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6shks"
Jul 26 15:51:30.449648 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:30.449620    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:30.550091 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:30.550024    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6shks"]
Jul 26 15:51:30.554070 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:51:30.554036    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod212d7a6b_fb00_48bf_88a1_c5d617abe2cb.slice/crio-041238a9461e4419c2e00ef583fafec10637697ee30228bc94f71d3e20e804ae WatchSource:0}: Error finding container 041238a9461e4419c2e00ef583fafec10637697ee30228bc94f71d3e20e804ae: Status 404 returned error can't find the container with id 041238a9461e4419c2e00ef583fafec10637697ee30228bc94f71d3e20e804ae
Jul 26 15:51:30.899523 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:30.899446    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6shks" event={"ID":"212d7a6b-fb00-48bf-88a1-c5d617abe2cb","Type":"ContainerStarted","Data":"041238a9461e4419c2e00ef583fafec10637697ee30228bc94f71d3e20e804ae"}
Jul 26 15:51:30.950288 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:30.950249    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:31.049160 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:31.049122    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:31.149750 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:31.149496    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:32.392837 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:32.392781    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:32.443993 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:32.443958    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:33.184887 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:33.184851    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:33.297557 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:33.297526    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:33.420939 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:33.420906    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:33.500294 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:33.500160    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:33.544644 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:33.543358    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-pzvmc"]
Jul 26 15:51:33.569108 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:33.561091    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pzvmc"
Jul 26 15:51:33.570472 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:33.569684    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-qqbbn\""
Jul 26 15:51:33.570472 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:33.569712    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 26 15:51:33.570472 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:33.569985    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 26 15:51:33.570472 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:33.570011    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 26 15:51:33.570472 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:33.570195    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 26 15:51:33.571010 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:33.570945    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 26 15:51:33.571010 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:33.570961    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 26 15:51:33.575478 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:33.575395    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:33.610416 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:33.610259    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0ed65756-c097-4851-9612-6a63e2cec317-node-exporter-tls\") pod \"node-exporter-pzvmc\" (UID: \"0ed65756-c097-4851-9612-6a63e2cec317\") " pod="openshift-monitoring/node-exporter-pzvmc"
Jul 26 15:51:33.610416 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:33.610297    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0ed65756-c097-4851-9612-6a63e2cec317-root\") pod \"node-exporter-pzvmc\" (UID: \"0ed65756-c097-4851-9612-6a63e2cec317\") " pod="openshift-monitoring/node-exporter-pzvmc"
Jul 26 15:51:33.610416 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:33.610351    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ed65756-c097-4851-9612-6a63e2cec317-metrics-client-ca\") pod \"node-exporter-pzvmc\" (UID: \"0ed65756-c097-4851-9612-6a63e2cec317\") " pod="openshift-monitoring/node-exporter-pzvmc"
Jul 26 15:51:33.610671 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:33.610415    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0ed65756-c097-4851-9612-6a63e2cec317-node-exporter-wtmp\") pod \"node-exporter-pzvmc\" (UID: \"0ed65756-c097-4851-9612-6a63e2cec317\") " pod="openshift-monitoring/node-exporter-pzvmc"
Jul 26 15:51:33.610671 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:33.610460    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0ed65756-c097-4851-9612-6a63e2cec317-node-exporter-accelerators-collector-config\") pod \"node-exporter-pzvmc\" (UID: \"0ed65756-c097-4851-9612-6a63e2cec317\") " pod="openshift-monitoring/node-exporter-pzvmc"
Jul 26 15:51:33.610671 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:33.610498    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0ed65756-c097-4851-9612-6a63e2cec317-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pzvmc\" (UID: \"0ed65756-c097-4851-9612-6a63e2cec317\") " pod="openshift-monitoring/node-exporter-pzvmc"
Jul 26 15:51:33.610671 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:33.610529    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd2tw\" (UniqueName: \"kubernetes.io/projected/0ed65756-c097-4851-9612-6a63e2cec317-kube-api-access-hd2tw\") pod \"node-exporter-pzvmc\" (UID: \"0ed65756-c097-4851-9612-6a63e2cec317\") " pod="openshift-monitoring/node-exporter-pzvmc"
Jul 26 15:51:33.610671 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:33.610559    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0ed65756-c097-4851-9612-6a63e2cec317-sys\") pod \"node-exporter-pzvmc\" (UID: \"0ed65756-c097-4851-9612-6a63e2cec317\") " pod="openshift-monitoring/node-exporter-pzvmc"
Jul 26 15:51:33.610671 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:33.610582    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0ed65756-c097-4851-9612-6a63e2cec317-node-exporter-textfile\") pod \"node-exporter-pzvmc\" (UID: \"0ed65756-c097-4851-9612-6a63e2cec317\") " pod="openshift-monitoring/node-exporter-pzvmc"
Jul 26 15:51:33.642779 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:33.642714    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:33.711649 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:33.711297    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0ed65756-c097-4851-9612-6a63e2cec317-node-exporter-tls\") pod \"node-exporter-pzvmc\" (UID: \"0ed65756-c097-4851-9612-6a63e2cec317\") " pod="openshift-monitoring/node-exporter-pzvmc"
Jul 26 15:51:33.711649 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:33.711339    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0ed65756-c097-4851-9612-6a63e2cec317-root\") pod \"node-exporter-pzvmc\" (UID: \"0ed65756-c097-4851-9612-6a63e2cec317\") " pod="openshift-monitoring/node-exporter-pzvmc"
Jul 26 15:51:33.711649 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:33.711371    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ed65756-c097-4851-9612-6a63e2cec317-metrics-client-ca\") pod \"node-exporter-pzvmc\" (UID: \"0ed65756-c097-4851-9612-6a63e2cec317\") " pod="openshift-monitoring/node-exporter-pzvmc"
Jul 26 15:51:33.711649 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:33.711399    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0ed65756-c097-4851-9612-6a63e2cec317-node-exporter-wtmp\") pod \"node-exporter-pzvmc\" (UID: \"0ed65756-c097-4851-9612-6a63e2cec317\") " pod="openshift-monitoring/node-exporter-pzvmc"
Jul 26 15:51:33.711649 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:33.711430    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0ed65756-c097-4851-9612-6a63e2cec317-node-exporter-accelerators-collector-config\") pod \"node-exporter-pzvmc\" (UID: \"0ed65756-c097-4851-9612-6a63e2cec317\") " pod="openshift-monitoring/node-exporter-pzvmc"
Jul 26 15:51:33.711649 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:51:33.711462    2563 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 26 15:51:33.711649 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:51:33.711535    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ed65756-c097-4851-9612-6a63e2cec317-node-exporter-tls podName:0ed65756-c097-4851-9612-6a63e2cec317 nodeName:}" failed. No retries permitted until 2026-07-26 15:51:34.211514044 +0000 UTC m=+86.315739846 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/0ed65756-c097-4851-9612-6a63e2cec317-node-exporter-tls") pod "node-exporter-pzvmc" (UID: "0ed65756-c097-4851-9612-6a63e2cec317") : secret "node-exporter-tls" not found
Jul 26 15:51:33.712168 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:33.711464    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0ed65756-c097-4851-9612-6a63e2cec317-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pzvmc\" (UID: \"0ed65756-c097-4851-9612-6a63e2cec317\") " pod="openshift-monitoring/node-exporter-pzvmc"
Jul 26 15:51:33.712168 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:33.711834    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd2tw\" (UniqueName: \"kubernetes.io/projected/0ed65756-c097-4851-9612-6a63e2cec317-kube-api-access-hd2tw\") pod \"node-exporter-pzvmc\" (UID: \"0ed65756-c097-4851-9612-6a63e2cec317\") " pod="openshift-monitoring/node-exporter-pzvmc"
Jul 26 15:51:33.712168 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:33.711868    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0ed65756-c097-4851-9612-6a63e2cec317-sys\") pod \"node-exporter-pzvmc\" (UID: \"0ed65756-c097-4851-9612-6a63e2cec317\") " pod="openshift-monitoring/node-exporter-pzvmc"
Jul 26 15:51:33.712168 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:33.711893    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0ed65756-c097-4851-9612-6a63e2cec317-node-exporter-textfile\") pod \"node-exporter-pzvmc\" (UID: \"0ed65756-c097-4851-9612-6a63e2cec317\") " pod="openshift-monitoring/node-exporter-pzvmc"
Jul 26 15:51:33.712375 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:33.712185    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0ed65756-c097-4851-9612-6a63e2cec317-node-exporter-textfile\") pod \"node-exporter-pzvmc\" (UID: \"0ed65756-c097-4851-9612-6a63e2cec317\") " pod="openshift-monitoring/node-exporter-pzvmc"
Jul 26 15:51:33.712375 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:33.712213    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0ed65756-c097-4851-9612-6a63e2cec317-node-exporter-wtmp\") pod \"node-exporter-pzvmc\" (UID: \"0ed65756-c097-4851-9612-6a63e2cec317\") " pod="openshift-monitoring/node-exporter-pzvmc"
Jul 26 15:51:33.712375 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:33.712269    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0ed65756-c097-4851-9612-6a63e2cec317-sys\") pod \"node-exporter-pzvmc\" (UID: \"0ed65756-c097-4851-9612-6a63e2cec317\") " pod="openshift-monitoring/node-exporter-pzvmc"
Jul 26 15:51:33.712375 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:33.712313    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0ed65756-c097-4851-9612-6a63e2cec317-root\") pod \"node-exporter-pzvmc\" (UID: \"0ed65756-c097-4851-9612-6a63e2cec317\") " pod="openshift-monitoring/node-exporter-pzvmc"
Jul 26 15:51:33.713306 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:33.712982    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0ed65756-c097-4851-9612-6a63e2cec317-node-exporter-accelerators-collector-config\") pod \"node-exporter-pzvmc\" (UID: \"0ed65756-c097-4851-9612-6a63e2cec317\") " pod="openshift-monitoring/node-exporter-pzvmc"
Jul 26 15:51:33.713306 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:33.713190    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ed65756-c097-4851-9612-6a63e2cec317-metrics-client-ca\") pod \"node-exporter-pzvmc\" (UID: \"0ed65756-c097-4851-9612-6a63e2cec317\") " pod="openshift-monitoring/node-exporter-pzvmc"
Jul 26 15:51:33.720789 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:33.720737    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0ed65756-c097-4851-9612-6a63e2cec317-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pzvmc\" (UID: \"0ed65756-c097-4851-9612-6a63e2cec317\") " pod="openshift-monitoring/node-exporter-pzvmc"
Jul 26 15:51:33.729057 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:33.729001    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd2tw\" (UniqueName: \"kubernetes.io/projected/0ed65756-c097-4851-9612-6a63e2cec317-kube-api-access-hd2tw\") pod \"node-exporter-pzvmc\" (UID: \"0ed65756-c097-4851-9612-6a63e2cec317\") " pod="openshift-monitoring/node-exporter-pzvmc"
Jul 26 15:51:33.731412 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:33.731349    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:33.832323 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:33.832239    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:33.899780 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:33.899017    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:34.019897 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:34.019848    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:34.119657 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:34.119576    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:34.216350 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:34.216257    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0ed65756-c097-4851-9612-6a63e2cec317-node-exporter-tls\") pod \"node-exporter-pzvmc\" (UID: \"0ed65756-c097-4851-9612-6a63e2cec317\") " pod="openshift-monitoring/node-exporter-pzvmc"
Jul 26 15:51:34.218874 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:34.218853    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0ed65756-c097-4851-9612-6a63e2cec317-node-exporter-tls\") pod \"node-exporter-pzvmc\" (UID: \"0ed65756-c097-4851-9612-6a63e2cec317\") " pod="openshift-monitoring/node-exporter-pzvmc"
Jul 26 15:51:34.439153 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:34.439079    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:34.481861 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:34.481779    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pzvmc"
Jul 26 15:51:34.542453 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:34.542373    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:34.545280 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:34.545253    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 26 15:51:34.551215 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:34.551190    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 26 15:51:34.553785 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:34.553701    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 26 15:51:34.553785 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:34.553750    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 26 15:51:34.553785 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:34.553781    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 26 15:51:34.553999 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:34.553969    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 26 15:51:34.554058 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:34.554018    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 26 15:51:34.554756 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:34.554289    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 26 15:51:34.554756 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:34.554519    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 26 15:51:34.554756 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:34.554753    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 26 15:51:34.554980 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:34.554961    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 26 15:51:34.555039 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:34.555020    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-wtbsv\""
Jul 26 15:51:34.573523 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:34.573498    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 26 15:51:34.619839 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:34.619809    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/32649266-3ac3-4f8d-8d02-0ea5b3462e41-config-out\") pod \"alertmanager-main-0\" (UID: \"32649266-3ac3-4f8d-8d02-0ea5b3462e41\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 26 15:51:34.619839 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:34.619838    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/32649266-3ac3-4f8d-8d02-0ea5b3462e41-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"32649266-3ac3-4f8d-8d02-0ea5b3462e41\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 26 15:51:34.620113 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:34.619867    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/32649266-3ac3-4f8d-8d02-0ea5b3462e41-tls-assets\") pod \"alertmanager-main-0\" (UID: \"32649266-3ac3-4f8d-8d02-0ea5b3462e41\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 26 15:51:34.620113 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:34.619890    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/32649266-3ac3-4f8d-8d02-0ea5b3462e41-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"32649266-3ac3-4f8d-8d02-0ea5b3462e41\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 26 15:51:34.620113 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:34.619911    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/32649266-3ac3-4f8d-8d02-0ea5b3462e41-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"32649266-3ac3-4f8d-8d02-0ea5b3462e41\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 26 15:51:34.620113 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:34.619929    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/32649266-3ac3-4f8d-8d02-0ea5b3462e41-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"32649266-3ac3-4f8d-8d02-0ea5b3462e41\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 26 15:51:34.620113 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:34.619957    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/32649266-3ac3-4f8d-8d02-0ea5b3462e41-web-config\") pod \"alertmanager-main-0\" (UID: \"32649266-3ac3-4f8d-8d02-0ea5b3462e41\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 26 15:51:34.620113 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:34.619973    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32649266-3ac3-4f8d-8d02-0ea5b3462e41-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"32649266-3ac3-4f8d-8d02-0ea5b3462e41\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 26 15:51:34.620113 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:34.620000    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32649266-3ac3-4f8d-8d02-0ea5b3462e41-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"32649266-3ac3-4f8d-8d02-0ea5b3462e41\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 26 15:51:34.620113 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:34.620025    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ggxw\" (UniqueName: \"kubernetes.io/projected/32649266-3ac3-4f8d-8d02-0ea5b3462e41-kube-api-access-9ggxw\") pod \"alertmanager-main-0\" (UID: \"32649266-3ac3-4f8d-8d02-0ea5b3462e41\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 26 15:51:34.620113 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:34.620089    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/32649266-3ac3-4f8d-8d02-0ea5b3462e41-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"32649266-3ac3-4f8d-8d02-0ea5b3462e41\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 26 15:51:34.620642 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:34.620121    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/32649266-3ac3-4f8d-8d02-0ea5b3462e41-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"32649266-3ac3-4f8d-8d02-0ea5b3462e41\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 26 15:51:34.620642 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:34.620148    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/32649266-3ac3-4f8d-8d02-0ea5b3462e41-config-volume\") pod \"alertmanager-main-0\" (UID: \"32649266-3ac3-4f8d-8d02-0ea5b3462e41\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 26 15:51:34.720951 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:34.720856    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/32649266-3ac3-4f8d-8d02-0ea5b3462e41-tls-assets\") pod \"alertmanager-main-0\" (UID: \"32649266-3ac3-4f8d-8d02-0ea5b3462e41\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 26 15:51:34.720951 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:34.720919    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/32649266-3ac3-4f8d-8d02-0ea5b3462e41-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"32649266-3ac3-4f8d-8d02-0ea5b3462e41\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 26 15:51:34.721153 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:34.720971    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/32649266-3ac3-4f8d-8d02-0ea5b3462e41-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"32649266-3ac3-4f8d-8d02-0ea5b3462e41\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 26 15:51:34.721153 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:34.721005    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/32649266-3ac3-4f8d-8d02-0ea5b3462e41-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"32649266-3ac3-4f8d-8d02-0ea5b3462e41\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 26 15:51:34.721153 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:34.721042    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/32649266-3ac3-4f8d-8d02-0ea5b3462e41-web-config\") pod \"alertmanager-main-0\" (UID: \"32649266-3ac3-4f8d-8d02-0ea5b3462e41\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 26 15:51:34.721153 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:34.721070    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32649266-3ac3-4f8d-8d02-0ea5b3462e41-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"32649266-3ac3-4f8d-8d02-0ea5b3462e41\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 26 15:51:34.721153 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:34.721103    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32649266-3ac3-4f8d-8d02-0ea5b3462e41-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"32649266-3ac3-4f8d-8d02-0ea5b3462e41\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 26 15:51:34.721153 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:34.721136    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ggxw\" (UniqueName: \"kubernetes.io/projected/32649266-3ac3-4f8d-8d02-0ea5b3462e41-kube-api-access-9ggxw\") pod \"alertmanager-main-0\" (UID: \"32649266-3ac3-4f8d-8d02-0ea5b3462e41\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 26 15:51:34.721414 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:34.721203    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/32649266-3ac3-4f8d-8d02-0ea5b3462e41-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"32649266-3ac3-4f8d-8d02-0ea5b3462e41\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 26 15:51:34.721414 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:34.721237    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/32649266-3ac3-4f8d-8d02-0ea5b3462e41-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"32649266-3ac3-4f8d-8d02-0ea5b3462e41\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 26 15:51:34.721414 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:34.721268    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/32649266-3ac3-4f8d-8d02-0ea5b3462e41-config-volume\") pod \"alertmanager-main-0\" (UID: \"32649266-3ac3-4f8d-8d02-0ea5b3462e41\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 26 15:51:34.721414 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:34.721301    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/32649266-3ac3-4f8d-8d02-0ea5b3462e41-config-out\") pod \"alertmanager-main-0\" (UID: \"32649266-3ac3-4f8d-8d02-0ea5b3462e41\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 26 15:51:34.721414 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:34.721328    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/32649266-3ac3-4f8d-8d02-0ea5b3462e41-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"32649266-3ac3-4f8d-8d02-0ea5b3462e41\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 26 15:51:34.723236 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:51:34.721749    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/32649266-3ac3-4f8d-8d02-0ea5b3462e41-alertmanager-trusted-ca-bundle podName:32649266-3ac3-4f8d-8d02-0ea5b3462e41 nodeName:}" failed. No retries permitted until 2026-07-26 15:51:35.221726963 +0000 UTC m=+87.325952751 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/32649266-3ac3-4f8d-8d02-0ea5b3462e41-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "32649266-3ac3-4f8d-8d02-0ea5b3462e41") : configmap references non-existent config key: ca-bundle.crt
Jul 26 15:51:34.723236 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:34.722061    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32649266-3ac3-4f8d-8d02-0ea5b3462e41-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"32649266-3ac3-4f8d-8d02-0ea5b3462e41\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 26 15:51:34.723236 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:34.722553    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/32649266-3ac3-4f8d-8d02-0ea5b3462e41-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"32649266-3ac3-4f8d-8d02-0ea5b3462e41\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 26 15:51:34.723236 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:51:34.722653    2563 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Jul 26 15:51:34.723236 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:51:34.722716    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32649266-3ac3-4f8d-8d02-0ea5b3462e41-secret-alertmanager-main-tls podName:32649266-3ac3-4f8d-8d02-0ea5b3462e41 nodeName:}" failed. No retries permitted until 2026-07-26 15:51:35.222697308 +0000 UTC m=+87.326923100 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/32649266-3ac3-4f8d-8d02-0ea5b3462e41-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "32649266-3ac3-4f8d-8d02-0ea5b3462e41") : secret "alertmanager-main-tls" not found
Jul 26 15:51:34.724268 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:34.724242    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/32649266-3ac3-4f8d-8d02-0ea5b3462e41-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"32649266-3ac3-4f8d-8d02-0ea5b3462e41\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 26 15:51:34.724971 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:34.724931    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/32649266-3ac3-4f8d-8d02-0ea5b3462e41-tls-assets\") pod \"alertmanager-main-0\" (UID: \"32649266-3ac3-4f8d-8d02-0ea5b3462e41\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 26 15:51:34.726548 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:34.726246    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/32649266-3ac3-4f8d-8d02-0ea5b3462e41-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"32649266-3ac3-4f8d-8d02-0ea5b3462e41\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 26 15:51:34.726548 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:34.726507    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/32649266-3ac3-4f8d-8d02-0ea5b3462e41-web-config\") pod \"alertmanager-main-0\" (UID: \"32649266-3ac3-4f8d-8d02-0ea5b3462e41\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 26 15:51:34.727131 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:34.727106    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/32649266-3ac3-4f8d-8d02-0ea5b3462e41-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"32649266-3ac3-4f8d-8d02-0ea5b3462e41\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 26 15:51:34.727582 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:34.727537    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/32649266-3ac3-4f8d-8d02-0ea5b3462e41-config-out\") pod \"alertmanager-main-0\" (UID: \"32649266-3ac3-4f8d-8d02-0ea5b3462e41\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 26 15:51:34.727820 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:34.727799    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/32649266-3ac3-4f8d-8d02-0ea5b3462e41-config-volume\") pod \"alertmanager-main-0\" (UID: \"32649266-3ac3-4f8d-8d02-0ea5b3462e41\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 26 15:51:34.728630 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:34.728608    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/32649266-3ac3-4f8d-8d02-0ea5b3462e41-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"32649266-3ac3-4f8d-8d02-0ea5b3462e41\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 26 15:51:34.739021 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:34.738994    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ggxw\" (UniqueName: \"kubernetes.io/projected/32649266-3ac3-4f8d-8d02-0ea5b3462e41-kube-api-access-9ggxw\") pod \"alertmanager-main-0\" (UID: \"32649266-3ac3-4f8d-8d02-0ea5b3462e41\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 26 15:51:34.749455 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:34.749414    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:34.841182 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:34.841148    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:35.127131 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:35.127089    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:35.226583 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:35.226545    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/32649266-3ac3-4f8d-8d02-0ea5b3462e41-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"32649266-3ac3-4f8d-8d02-0ea5b3462e41\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 26 15:51:35.226811 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:35.226619    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32649266-3ac3-4f8d-8d02-0ea5b3462e41-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"32649266-3ac3-4f8d-8d02-0ea5b3462e41\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 26 15:51:35.227347 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:35.227322    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32649266-3ac3-4f8d-8d02-0ea5b3462e41-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"32649266-3ac3-4f8d-8d02-0ea5b3462e41\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 26 15:51:35.228945 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:35.228925    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/32649266-3ac3-4f8d-8d02-0ea5b3462e41-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"32649266-3ac3-4f8d-8d02-0ea5b3462e41\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 26 15:51:35.464996 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:35.464908    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 26 15:51:35.562652 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:51:35.562527    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ed65756_c097_4851_9612_6a63e2cec317.slice/crio-4c3eada64bcd97ad9405c165f2f5019e11ac5c505ac175b5055b01053507ba4a WatchSource:0}: Error finding container 4c3eada64bcd97ad9405c165f2f5019e11ac5c505ac175b5055b01053507ba4a: Status 404 returned error can't find the container with id 4c3eada64bcd97ad9405c165f2f5019e11ac5c505ac175b5055b01053507ba4a
Jul 26 15:51:35.593510 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:35.593480    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:35.695033 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:35.695002    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:35.715947 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:35.715775    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 26 15:51:35.718831 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:51:35.718809    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32649266_3ac3_4f8d_8d02_0ea5b3462e41.slice/crio-1a377d9595e5dff846a95f093632a36706e00e1d7d17df4f9fb34687a6732ba3 WatchSource:0}: Error finding container 1a377d9595e5dff846a95f093632a36706e00e1d7d17df4f9fb34687a6732ba3: Status 404 returned error can't find the container with id 1a377d9595e5dff846a95f093632a36706e00e1d7d17df4f9fb34687a6732ba3
Jul 26 15:51:35.927154 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:35.927112    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6shks" event={"ID":"212d7a6b-fb00-48bf-88a1-c5d617abe2cb","Type":"ContainerStarted","Data":"8a86ee1bdaa2ab7aecd2aa98ce58f0c383b080cce0ddd9deb696d035fb1f2fd1"}
Jul 26 15:51:35.928429 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:35.928396    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"32649266-3ac3-4f8d-8d02-0ea5b3462e41","Type":"ContainerStarted","Data":"1a377d9595e5dff846a95f093632a36706e00e1d7d17df4f9fb34687a6732ba3"}
Jul 26 15:51:35.929720 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:35.929691    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pzvmc" event={"ID":"0ed65756-c097-4851-9612-6a63e2cec317","Type":"ContainerStarted","Data":"4c3eada64bcd97ad9405c165f2f5019e11ac5c505ac175b5055b01053507ba4a"}
Jul 26 15:51:36.386984 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:36.386903    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:36.459024 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:36.458986    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:36.531190 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:36.531159    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:36.851631 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:36.851598    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:36.935599 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:36.935558    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pzvmc" event={"ID":"0ed65756-c097-4851-9612-6a63e2cec317","Type":"ContainerStarted","Data":"2476175e2c9b80de3a6082301e57776d2ec4afc052e02d97ebb109e3127954a8"}
Jul 26 15:51:36.949053 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:36.948968    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:37.941204 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:37.941164    2563 generic.go:356] "Generic (PLEG): container finished" podID="32649266-3ac3-4f8d-8d02-0ea5b3462e41" containerID="645065a63dfbe92c096216ba008a1efc7426ed8f60c4b0d35d6be0717fee487c" exitCode=0
Jul 26 15:51:37.941689 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:37.941249    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"32649266-3ac3-4f8d-8d02-0ea5b3462e41","Type":"ContainerDied","Data":"645065a63dfbe92c096216ba008a1efc7426ed8f60c4b0d35d6be0717fee487c"}
Jul 26 15:51:37.943280 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:37.943258    2563 generic.go:356] "Generic (PLEG): container finished" podID="0ed65756-c097-4851-9612-6a63e2cec317" containerID="2476175e2c9b80de3a6082301e57776d2ec4afc052e02d97ebb109e3127954a8" exitCode=0
Jul 26 15:51:37.943381 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:37.943307    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pzvmc" event={"ID":"0ed65756-c097-4851-9612-6a63e2cec317","Type":"ContainerDied","Data":"2476175e2c9b80de3a6082301e57776d2ec4afc052e02d97ebb109e3127954a8"}
Jul 26 15:51:38.236649 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:38.236600    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:38.253372 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:38.253342    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-94stp"]
Jul 26 15:51:38.275034 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:38.275006    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-94stp"]
Jul 26 15:51:38.275181 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:38.275113    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-94stp"
Jul 26 15:51:38.277864 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:38.277829    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-2rc62\""
Jul 26 15:51:38.277864 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:38.277838    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 26 15:51:38.299170 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:38.299136    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:38.355902 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:38.355870    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5cd15409-1edd-41c2-bc75-1a5851945440-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-94stp\" (UID: \"5cd15409-1edd-41c2-bc75-1a5851945440\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-94stp"
Jul 26 15:51:38.356072 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:38.355949    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5fh4\" (UniqueName: \"kubernetes.io/projected/5cd15409-1edd-41c2-bc75-1a5851945440-kube-api-access-g5fh4\") pod \"monitoring-plugin-68b7fc9574-94stp\" (UID: \"5cd15409-1edd-41c2-bc75-1a5851945440\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-94stp"
Jul 26 15:51:38.457018 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:38.456982    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5cd15409-1edd-41c2-bc75-1a5851945440-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-94stp\" (UID: \"5cd15409-1edd-41c2-bc75-1a5851945440\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-94stp"
Jul 26 15:51:38.457214 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:38.457086    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5fh4\" (UniqueName: \"kubernetes.io/projected/5cd15409-1edd-41c2-bc75-1a5851945440-kube-api-access-g5fh4\") pod \"monitoring-plugin-68b7fc9574-94stp\" (UID: \"5cd15409-1edd-41c2-bc75-1a5851945440\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-94stp"
Jul 26 15:51:38.457214 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:51:38.457138    2563 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Jul 26 15:51:38.457333 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:51:38.457226    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cd15409-1edd-41c2-bc75-1a5851945440-monitoring-plugin-cert podName:5cd15409-1edd-41c2-bc75-1a5851945440 nodeName:}" failed. No retries permitted until 2026-07-26 15:51:38.957206203 +0000 UTC m=+91.061431991 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/5cd15409-1edd-41c2-bc75-1a5851945440-monitoring-plugin-cert") pod "monitoring-plugin-68b7fc9574-94stp" (UID: "5cd15409-1edd-41c2-bc75-1a5851945440") : secret "monitoring-plugin-cert" not found
Jul 26 15:51:38.467814 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:38.467746    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5fh4\" (UniqueName: \"kubernetes.io/projected/5cd15409-1edd-41c2-bc75-1a5851945440-kube-api-access-g5fh4\") pod \"monitoring-plugin-68b7fc9574-94stp\" (UID: \"5cd15409-1edd-41c2-bc75-1a5851945440\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-94stp"
Jul 26 15:51:38.961229 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:38.961187    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5cd15409-1edd-41c2-bc75-1a5851945440-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-94stp\" (UID: \"5cd15409-1edd-41c2-bc75-1a5851945440\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-94stp"
Jul 26 15:51:38.961721 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:51:38.961399    2563 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Jul 26 15:51:38.961721 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:51:38.961484    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cd15409-1edd-41c2-bc75-1a5851945440-monitoring-plugin-cert podName:5cd15409-1edd-41c2-bc75-1a5851945440 nodeName:}" failed. No retries permitted until 2026-07-26 15:51:39.961462471 +0000 UTC m=+92.065688261 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/5cd15409-1edd-41c2-bc75-1a5851945440-monitoring-plugin-cert") pod "monitoring-plugin-68b7fc9574-94stp" (UID: "5cd15409-1edd-41c2-bc75-1a5851945440") : secret "monitoring-plugin-cert" not found
Jul 26 15:51:39.277593 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.277545    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:39.635730 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.635642    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:39.643891 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.643857    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 26 15:51:39.648086 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.648061    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 26 15:51:39.650817 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.650788    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 26 15:51:39.650944 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.650822    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 26 15:51:39.650944 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.650844    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 26 15:51:39.651106 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.651090    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 26 15:51:39.651311 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.651193    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-eitsbs022jiab\""
Jul 26 15:51:39.651311 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.651254    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-gshvb\""
Jul 26 15:51:39.651311 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.651261    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 26 15:51:39.651517 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.651497    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 26 15:51:39.651570 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.651535    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 26 15:51:39.651570 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.651545    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 26 15:51:39.651671 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.651598    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 26 15:51:39.651890 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.651802    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 26 15:51:39.651890 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.651809    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 26 15:51:39.651890 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.651868    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 26 15:51:39.651890 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.651868    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 26 15:51:39.652144 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.651840    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 26 15:51:39.653125 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.653101    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 26 15:51:39.667929 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.667908    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 26 15:51:39.766412 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.766385    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/645f034c-adf9-4a52-a059-2fa1d3988867-web-config\") pod \"prometheus-k8s-0\" (UID: \"645f034c-adf9-4a52-a059-2fa1d3988867\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 26 15:51:39.766524 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.766424    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/645f034c-adf9-4a52-a059-2fa1d3988867-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"645f034c-adf9-4a52-a059-2fa1d3988867\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 26 15:51:39.766524 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.766442    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/645f034c-adf9-4a52-a059-2fa1d3988867-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"645f034c-adf9-4a52-a059-2fa1d3988867\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 26 15:51:39.766524 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.766462    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24gcw\" (UniqueName: \"kubernetes.io/projected/645f034c-adf9-4a52-a059-2fa1d3988867-kube-api-access-24gcw\") pod \"prometheus-k8s-0\" (UID: \"645f034c-adf9-4a52-a059-2fa1d3988867\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 26 15:51:39.766625 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.766542    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/645f034c-adf9-4a52-a059-2fa1d3988867-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"645f034c-adf9-4a52-a059-2fa1d3988867\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 26 15:51:39.766625 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.766566    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/645f034c-adf9-4a52-a059-2fa1d3988867-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"645f034c-adf9-4a52-a059-2fa1d3988867\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 26 15:51:39.766625 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.766589    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/645f034c-adf9-4a52-a059-2fa1d3988867-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"645f034c-adf9-4a52-a059-2fa1d3988867\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 26 15:51:39.766714 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.766622    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/645f034c-adf9-4a52-a059-2fa1d3988867-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"645f034c-adf9-4a52-a059-2fa1d3988867\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 26 15:51:39.766714 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.766654    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/645f034c-adf9-4a52-a059-2fa1d3988867-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"645f034c-adf9-4a52-a059-2fa1d3988867\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 26 15:51:39.766714 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.766683    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/645f034c-adf9-4a52-a059-2fa1d3988867-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"645f034c-adf9-4a52-a059-2fa1d3988867\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 26 15:51:39.766714 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.766700    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/645f034c-adf9-4a52-a059-2fa1d3988867-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"645f034c-adf9-4a52-a059-2fa1d3988867\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 26 15:51:39.766902 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.766719    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/645f034c-adf9-4a52-a059-2fa1d3988867-config-out\") pod \"prometheus-k8s-0\" (UID: \"645f034c-adf9-4a52-a059-2fa1d3988867\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 26 15:51:39.766902 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.766739    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/645f034c-adf9-4a52-a059-2fa1d3988867-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"645f034c-adf9-4a52-a059-2fa1d3988867\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 26 15:51:39.766902 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.766805    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/645f034c-adf9-4a52-a059-2fa1d3988867-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"645f034c-adf9-4a52-a059-2fa1d3988867\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 26 15:51:39.766902 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.766828    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/645f034c-adf9-4a52-a059-2fa1d3988867-config\") pod \"prometheus-k8s-0\" (UID: \"645f034c-adf9-4a52-a059-2fa1d3988867\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 26 15:51:39.766902 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.766856    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/645f034c-adf9-4a52-a059-2fa1d3988867-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"645f034c-adf9-4a52-a059-2fa1d3988867\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 26 15:51:39.766902 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.766891    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/645f034c-adf9-4a52-a059-2fa1d3988867-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"645f034c-adf9-4a52-a059-2fa1d3988867\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 26 15:51:39.767149 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.766909    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/645f034c-adf9-4a52-a059-2fa1d3988867-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"645f034c-adf9-4a52-a059-2fa1d3988867\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 26 15:51:39.767149 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.766945    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/645f034c-adf9-4a52-a059-2fa1d3988867-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"645f034c-adf9-4a52-a059-2fa1d3988867\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 26 15:51:39.767149 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.766982    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/645f034c-adf9-4a52-a059-2fa1d3988867-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"645f034c-adf9-4a52-a059-2fa1d3988867\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 26 15:51:39.867677 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.867648    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/645f034c-adf9-4a52-a059-2fa1d3988867-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"645f034c-adf9-4a52-a059-2fa1d3988867\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 26 15:51:39.867786 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.867701    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/645f034c-adf9-4a52-a059-2fa1d3988867-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"645f034c-adf9-4a52-a059-2fa1d3988867\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 26 15:51:39.867786 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.867729    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/645f034c-adf9-4a52-a059-2fa1d3988867-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"645f034c-adf9-4a52-a059-2fa1d3988867\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 26 15:51:39.867786 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.867775    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/645f034c-adf9-4a52-a059-2fa1d3988867-config-out\") pod \"prometheus-k8s-0\" (UID: \"645f034c-adf9-4a52-a059-2fa1d3988867\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 26 15:51:39.867940 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.867889    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/645f034c-adf9-4a52-a059-2fa1d3988867-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"645f034c-adf9-4a52-a059-2fa1d3988867\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 26 15:51:39.867990 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.867937    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/645f034c-adf9-4a52-a059-2fa1d3988867-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"645f034c-adf9-4a52-a059-2fa1d3988867\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 26 15:51:39.867990 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.867971    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/645f034c-adf9-4a52-a059-2fa1d3988867-config\") pod \"prometheus-k8s-0\" (UID: \"645f034c-adf9-4a52-a059-2fa1d3988867\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 26 15:51:39.868083 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.868000    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/645f034c-adf9-4a52-a059-2fa1d3988867-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"645f034c-adf9-4a52-a059-2fa1d3988867\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 26 15:51:39.868083 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.868050    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/645f034c-adf9-4a52-a059-2fa1d3988867-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"645f034c-adf9-4a52-a059-2fa1d3988867\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 26 15:51:39.868233 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.868080    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/645f034c-adf9-4a52-a059-2fa1d3988867-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"645f034c-adf9-4a52-a059-2fa1d3988867\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 26 15:51:39.868233 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.868110    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/645f034c-adf9-4a52-a059-2fa1d3988867-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"645f034c-adf9-4a52-a059-2fa1d3988867\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 26 15:51:39.868233 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.868140    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/645f034c-adf9-4a52-a059-2fa1d3988867-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"645f034c-adf9-4a52-a059-2fa1d3988867\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 26 15:51:39.868233 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.868169    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/645f034c-adf9-4a52-a059-2fa1d3988867-web-config\") pod \"prometheus-k8s-0\" (UID: \"645f034c-adf9-4a52-a059-2fa1d3988867\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 26 15:51:39.868233 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.868207    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/645f034c-adf9-4a52-a059-2fa1d3988867-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"645f034c-adf9-4a52-a059-2fa1d3988867\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 26 15:51:39.868233 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.868229    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/645f034c-adf9-4a52-a059-2fa1d3988867-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"645f034c-adf9-4a52-a059-2fa1d3988867\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 26 15:51:39.868530 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.868261    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24gcw\" (UniqueName: \"kubernetes.io/projected/645f034c-adf9-4a52-a059-2fa1d3988867-kube-api-access-24gcw\") pod \"prometheus-k8s-0\" (UID: \"645f034c-adf9-4a52-a059-2fa1d3988867\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 26 15:51:39.868530 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.868299    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/645f034c-adf9-4a52-a059-2fa1d3988867-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"645f034c-adf9-4a52-a059-2fa1d3988867\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 26 15:51:39.868530 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.868311    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/645f034c-adf9-4a52-a059-2fa1d3988867-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"645f034c-adf9-4a52-a059-2fa1d3988867\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 26 15:51:39.868530 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.868336    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/645f034c-adf9-4a52-a059-2fa1d3988867-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"645f034c-adf9-4a52-a059-2fa1d3988867\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 26 15:51:39.868530 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.868367    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/645f034c-adf9-4a52-a059-2fa1d3988867-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"645f034c-adf9-4a52-a059-2fa1d3988867\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 26 15:51:39.870413 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.868978    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/645f034c-adf9-4a52-a059-2fa1d3988867-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"645f034c-adf9-4a52-a059-2fa1d3988867\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 26 15:51:39.870413 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.869074    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/645f034c-adf9-4a52-a059-2fa1d3988867-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"645f034c-adf9-4a52-a059-2fa1d3988867\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 26 15:51:39.870413 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.869116    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/645f034c-adf9-4a52-a059-2fa1d3988867-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"645f034c-adf9-4a52-a059-2fa1d3988867\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 26 15:51:39.870413 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.869434    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/645f034c-adf9-4a52-a059-2fa1d3988867-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"645f034c-adf9-4a52-a059-2fa1d3988867\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 26 15:51:39.870413 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.869870    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/645f034c-adf9-4a52-a059-2fa1d3988867-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"645f034c-adf9-4a52-a059-2fa1d3988867\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 26 15:51:39.870413 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.870362    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/645f034c-adf9-4a52-a059-2fa1d3988867-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"645f034c-adf9-4a52-a059-2fa1d3988867\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 26 15:51:39.870778 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.870739    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/645f034c-adf9-4a52-a059-2fa1d3988867-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"645f034c-adf9-4a52-a059-2fa1d3988867\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 26 15:51:39.871062 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.871008    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/645f034c-adf9-4a52-a059-2fa1d3988867-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"645f034c-adf9-4a52-a059-2fa1d3988867\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 26 15:51:39.871341 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.871311    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/645f034c-adf9-4a52-a059-2fa1d3988867-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"645f034c-adf9-4a52-a059-2fa1d3988867\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 26 15:51:39.872464 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.871951    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/645f034c-adf9-4a52-a059-2fa1d3988867-web-config\") pod \"prometheus-k8s-0\" (UID: \"645f034c-adf9-4a52-a059-2fa1d3988867\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 26 15:51:39.872464 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.872344    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/645f034c-adf9-4a52-a059-2fa1d3988867-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"645f034c-adf9-4a52-a059-2fa1d3988867\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 26 15:51:39.872464 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.872407    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/645f034c-adf9-4a52-a059-2fa1d3988867-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"645f034c-adf9-4a52-a059-2fa1d3988867\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 26 15:51:39.872692 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.872472    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/645f034c-adf9-4a52-a059-2fa1d3988867-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"645f034c-adf9-4a52-a059-2fa1d3988867\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 26 15:51:39.873095 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.873063    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/645f034c-adf9-4a52-a059-2fa1d3988867-config-out\") pod \"prometheus-k8s-0\" (UID: \"645f034c-adf9-4a52-a059-2fa1d3988867\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 26 15:51:39.873418 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.873399    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/645f034c-adf9-4a52-a059-2fa1d3988867-config\") pod \"prometheus-k8s-0\" (UID: \"645f034c-adf9-4a52-a059-2fa1d3988867\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 26 15:51:39.873500 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.873433    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/645f034c-adf9-4a52-a059-2fa1d3988867-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"645f034c-adf9-4a52-a059-2fa1d3988867\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 26 15:51:39.873624 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.873578    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/645f034c-adf9-4a52-a059-2fa1d3988867-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"645f034c-adf9-4a52-a059-2fa1d3988867\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 26 15:51:39.873624 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.873602    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/645f034c-adf9-4a52-a059-2fa1d3988867-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"645f034c-adf9-4a52-a059-2fa1d3988867\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 26 15:51:39.874000 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.873982    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/645f034c-adf9-4a52-a059-2fa1d3988867-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"645f034c-adf9-4a52-a059-2fa1d3988867\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 26 15:51:39.882150 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.882117    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24gcw\" (UniqueName: \"kubernetes.io/projected/645f034c-adf9-4a52-a059-2fa1d3988867-kube-api-access-24gcw\") pod \"prometheus-k8s-0\" (UID: \"645f034c-adf9-4a52-a059-2fa1d3988867\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 26 15:51:39.957289 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.957260    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6shks" event={"ID":"212d7a6b-fb00-48bf-88a1-c5d617abe2cb","Type":"ContainerStarted","Data":"a31f0350df6bf3113870c8b0f05260e914c73543989760d835809fa4e5301d9e"}
Jul 26 15:51:39.959032 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.959009    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 26 15:51:39.960281 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.960260    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"32649266-3ac3-4f8d-8d02-0ea5b3462e41","Type":"ContainerStarted","Data":"f2bde00866aca007bfaedb7f75d3fcd7f304739fb2d5d159c70c606761000d60"}
Jul 26 15:51:39.960389 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.960320    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"32649266-3ac3-4f8d-8d02-0ea5b3462e41","Type":"ContainerStarted","Data":"a58cdf94e5d22c79908a18f4688f457d614600f915d07712552b703a1ef34d4b"}
Jul 26 15:51:39.960389 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.960351    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"32649266-3ac3-4f8d-8d02-0ea5b3462e41","Type":"ContainerStarted","Data":"772c05b62c4d47a85a5688397e411ea1939820e727ce0b85dab9fa02fd46d51d"}
Jul 26 15:51:39.962251 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.962224    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pzvmc" event={"ID":"0ed65756-c097-4851-9612-6a63e2cec317","Type":"ContainerStarted","Data":"761b0fcdb8f71bb0204e5b71208b27969a6c7fa0b69620c4e5aa99ab97f6650a"}
Jul 26 15:51:39.962576 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.962280    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pzvmc" event={"ID":"0ed65756-c097-4851-9612-6a63e2cec317","Type":"ContainerStarted","Data":"729974f582e3ed636d1d0764987a1a5ed629e72d4e611f59166f7bc92feee43a"}
Jul 26 15:51:39.969672 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.969529    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5cd15409-1edd-41c2-bc75-1a5851945440-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-94stp\" (UID: \"5cd15409-1edd-41c2-bc75-1a5851945440\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-94stp"
Jul 26 15:51:39.971977 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.971955    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5cd15409-1edd-41c2-bc75-1a5851945440-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-94stp\" (UID: \"5cd15409-1edd-41c2-bc75-1a5851945440\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-94stp"
Jul 26 15:51:39.981424 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:39.981385    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-pzvmc" podStartSLOduration=5.949875771 podStartE2EDuration="6.981372809s" totalImagesPullingTime="949.458263ms" totalInitContainerRuntime="82.038775ms" isStatefulPod=true podCreationTimestamp="2026-07-26 15:51:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:51:39.980601953 +0000 UTC m=+92.084827786" watchObservedRunningTime="2026-07-26 15:51:39.981372809 +0000 UTC m=+92.085598618"
Jul 26 15:51:40.086519 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:40.086486    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-94stp"
Jul 26 15:51:40.110723 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:40.110690    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 26 15:51:40.115365 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:51:40.115342    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod645f034c_adf9_4a52_a059_2fa1d3988867.slice/crio-c3d9c4d4bc8b8aeb7b27b5aeb2bdd75abe736df90d5a83ca42e28fc3d89ce5fb WatchSource:0}: Error finding container c3d9c4d4bc8b8aeb7b27b5aeb2bdd75abe736df90d5a83ca42e28fc3d89ce5fb: Status 404 returned error can't find the container with id c3d9c4d4bc8b8aeb7b27b5aeb2bdd75abe736df90d5a83ca42e28fc3d89ce5fb
Jul 26 15:51:40.127229 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:40.127181    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:40.214087 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:40.213955    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-94stp"]
Jul 26 15:51:40.219956 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:40.219932    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:40.257676 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:51:40.257634    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cd15409_1edd_41c2_bc75_1a5851945440.slice/crio-25d12cd38c654137a3419f854eb6fdec946bcf432cbe981011d2414aee249bd0 WatchSource:0}: Error finding container 25d12cd38c654137a3419f854eb6fdec946bcf432cbe981011d2414aee249bd0: Status 404 returned error can't find the container with id 25d12cd38c654137a3419f854eb6fdec946bcf432cbe981011d2414aee249bd0
Jul 26 15:51:40.868166 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:40.868086    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:40.973349 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:40.973310    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"32649266-3ac3-4f8d-8d02-0ea5b3462e41","Type":"ContainerStarted","Data":"49abcc7a72a8986c617cfcc14a61d77b1bf9d789a2d65cf52bb22f30308312e7"}
Jul 26 15:51:40.973839 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:40.973400    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"32649266-3ac3-4f8d-8d02-0ea5b3462e41","Type":"ContainerStarted","Data":"22e5f2eaf1fd2c0e50a9beb4764c1ef0c0532ada40bc705c844c0d789017e12c"}
Jul 26 15:51:40.976254 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:40.976195    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-94stp" event={"ID":"5cd15409-1edd-41c2-bc75-1a5851945440","Type":"ContainerStarted","Data":"25d12cd38c654137a3419f854eb6fdec946bcf432cbe981011d2414aee249bd0"}
Jul 26 15:51:40.979280 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:40.979252    2563 generic.go:356] "Generic (PLEG): container finished" podID="645f034c-adf9-4a52-a059-2fa1d3988867" containerID="5d5ba628130bc51904c45f54fd8fe189523b2361f6a756f7605058d0f7ec2ba8" exitCode=0
Jul 26 15:51:40.980084 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:40.980060    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"645f034c-adf9-4a52-a059-2fa1d3988867","Type":"ContainerDied","Data":"5d5ba628130bc51904c45f54fd8fe189523b2361f6a756f7605058d0f7ec2ba8"}
Jul 26 15:51:40.980186 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:40.980131    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"645f034c-adf9-4a52-a059-2fa1d3988867","Type":"ContainerStarted","Data":"c3d9c4d4bc8b8aeb7b27b5aeb2bdd75abe736df90d5a83ca42e28fc3d89ce5fb"}
Jul 26 15:51:41.989156 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:41.989112    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"32649266-3ac3-4f8d-8d02-0ea5b3462e41","Type":"ContainerStarted","Data":"94bbf4a21c644b2581d59b5aaec440c6d5932619a8e7b9cb956fc539b7a1b28e"}
Jul 26 15:51:42.017655 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:42.017592    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.917912887 podStartE2EDuration="8.017569244s" totalImagesPullingTime="4.076847421s" totalInitContainerRuntime="22.808936ms" isStatefulPod=false podCreationTimestamp="2026-07-26 15:51:34 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:51:42.016212419 +0000 UTC m=+94.120438228" watchObservedRunningTime="2026-07-26 15:51:42.017569244 +0000 UTC m=+94.121795055"
Jul 26 15:51:45.002729 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:45.002698    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6shks" event={"ID":"212d7a6b-fb00-48bf-88a1-c5d617abe2cb","Type":"ContainerStarted","Data":"ab2d32db2748d40f89fcfad093a51ef7d785dc28a1c79fe68efbedc9f2c14861"}
Jul 26 15:51:45.004128 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:45.004106    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-94stp" event={"ID":"5cd15409-1edd-41c2-bc75-1a5851945440","Type":"ContainerStarted","Data":"c1284cb8db768674dc081dec6a1a719a67c21759951d77387a98bffd771a6ecb"}
Jul 26 15:51:45.006208 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:45.006178    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"645f034c-adf9-4a52-a059-2fa1d3988867","Type":"ContainerStarted","Data":"208166bd7cb7bfcb855481d08b64788d143539dda34e3c7285f669f513b8eb7c"}
Jul 26 15:51:45.020415 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:45.020381    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-94stp"
Jul 26 15:51:45.025775 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:45.025732    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-94stp"
Jul 26 15:51:45.036082 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:45.036021    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-94stp" podStartSLOduration=3.735235147 podStartE2EDuration="7.035999623s" totalImagesPullingTime="3.300764476s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:51:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:51:45.03539048 +0000 UTC m=+97.139616290" watchObservedRunningTime="2026-07-26 15:51:45.035999623 +0000 UTC m=+97.140225433"
Jul 26 15:51:45.052968 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:45.052908    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-6shks" podStartSLOduration=17.11978389 podStartE2EDuration="31.052892164s" totalImagesPullingTime="13.933108274s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 15:51:14 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:51:45.051915864 +0000 UTC m=+97.156141673" watchObservedRunningTime="2026-07-26 15:51:45.052892164 +0000 UTC m=+97.157117996"
Jul 26 15:51:46.012371 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:46.012334    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"645f034c-adf9-4a52-a059-2fa1d3988867","Type":"ContainerStarted","Data":"88f476b62b7e7d48d0be3d7a93ea1bf5cf5077bb9a396c52e1bb506e83bfec46"}
Jul 26 15:51:48.023849 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:48.023815    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"645f034c-adf9-4a52-a059-2fa1d3988867","Type":"ContainerStarted","Data":"5a7fe68e6718c35d960ec588b075685bd3b55786d969db816732b99ce47c4fdf"}
Jul 26 15:51:48.024355 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:48.023888    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"645f034c-adf9-4a52-a059-2fa1d3988867","Type":"ContainerStarted","Data":"067700a8cc61c524e88956b8d2b73527d52588fbd88fce5e2803847cd4bc6fba"}
Jul 26 15:51:48.024355 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:48.023928    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"645f034c-adf9-4a52-a059-2fa1d3988867","Type":"ContainerStarted","Data":"d1ede18fff1afab94329e373de4d53af2567710fef3f722e725cc28790bdd0a7"}
Jul 26 15:51:48.024355 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:48.023970    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"645f034c-adf9-4a52-a059-2fa1d3988867","Type":"ContainerStarted","Data":"42ae96ee8e0fabf5abf913ac411cda4239fcd923136507246ffdacd78d9b8b40"}
Jul 26 15:51:48.052895 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:48.052832    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.929660364 podStartE2EDuration="9.052811847s" totalImagesPullingTime="6.102219337s" totalInitContainerRuntime="20.932146ms" isStatefulPod=false podCreationTimestamp="2026-07-26 15:51:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:51:48.050602474 +0000 UTC m=+100.154828332" watchObservedRunningTime="2026-07-26 15:51:48.052811847 +0000 UTC m=+100.157037658"
Jul 26 15:51:48.160752 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:48.160720    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:48.261628 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:48.261595    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:49.067222 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:49.067170    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:49.168311 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:49.168264    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:49.274560 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:49.274526    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:49.867985 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:49.867954    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-7g75t"
Jul 26 15:51:49.959139 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:49.959104    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 26 15:51:50.096193 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:50.096160    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:50.206261 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:50.206176    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:51:50.297098 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:51:50.297067    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:52:01.138309 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:52:01.138274    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:52:01.194628 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:52:01.194599    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:52:01.245019 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:52:01.244988    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:52:03.156214 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:52:03.156183    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:52:03.209480 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:52:03.209447    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:52:10.050479 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:52:10.050442    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:52:10.151105 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:52:10.151066    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:52:10.211921 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:52:10.211876    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:52:39.960017 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:52:39.959968    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 26 15:52:39.978339 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:52:39.978315    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 26 15:52:40.231043 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:52:40.231017    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 26 15:52:40.978020 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:52:40.977989    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:52:44.693215 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:52:44.693143    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:52:44.745125 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:52:44.745096    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:52:49.532873 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:52:49.532840    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:52:49.633498 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:52:49.633469    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:52:59.561778 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:52:59.561743    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:53:01.403353 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:53:01.403272    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:53:01.456070 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:53:01.456034    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:53:04.538705 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:53:04.538669    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:53:04.691562 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:53:04.691527    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:53:04.741668 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:53:04.741640    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:53:05.305865 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:53:05.305835    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:53:05.356517 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:53:05.356487    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:53:15.431807 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:53:15.431749    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:53:15.485978 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:53:15.485948    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:53:15.540295 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:53:15.540269    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:53:18.162996 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:53:18.162965    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:53:18.263882 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:53:18.263848    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:53:53.124734 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:53:53.124701    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:53:53.220957 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:53:53.220925    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:54:19.829478 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:54:19.829444    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:54:20.660889 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:54:20.660852    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:54:23.433558 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:54:23.433506    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:54:23.534091 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:54:23.534057    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:54:25.779984 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:54:25.779948    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:54:25.882432 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:54:25.882395    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:54:25.997173 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:54:25.997142    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:54:26.100290 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:54:26.100199    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:54:27.979235 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:54:27.979202    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:54:28.179423 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:54:28.179392    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:54:30.425794 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:54:30.425703    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:54:30.475150 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:54:30.475108    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:54:34.234813 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:54:34.234783    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:54:34.335451 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:54:34.335421    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:54:36.380885 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:54:36.380850    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:54:38.713834 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:54:38.713790    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:54:38.814649 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:54:38.814615    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:54:45.456518 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:54:45.456473    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:54:45.520361 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:54:45.520330    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:54:47.263803 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:54:47.263754    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:55:00.097222 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:00.097186    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:55:00.194044 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:00.194008    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:55:00.715423 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:00.715390    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:55:00.977485 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:00.977439    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:55:01.073285 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:01.073249    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:55:01.136514 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:01.136482    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-745bd4f5cc-pxsh5"]
Jul 26 15:55:01.139826 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:01.139805    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-745bd4f5cc-pxsh5"
Jul 26 15:55:01.143376 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:01.143342    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 26 15:55:01.143645 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:01.143616    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 26 15:55:01.143723 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:01.143628    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-n4l27\""
Jul 26 15:55:01.143723 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:01.143672    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 26 15:55:01.143723 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:01.143678    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 26 15:55:01.157411 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:01.157385    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-745bd4f5cc-pxsh5"]
Jul 26 15:55:01.212682 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:01.212645    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/128af9ad-0db4-46f5-bee7-993eabdfc2f4-manifests\") pod \"opendatahub-operator-controller-manager-745bd4f5cc-pxsh5\" (UID: \"128af9ad-0db4-46f5-bee7-993eabdfc2f4\") " pod="opendatahub/opendatahub-operator-controller-manager-745bd4f5cc-pxsh5"
Jul 26 15:55:01.212877 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:01.212705    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-586wx\" (UniqueName: \"kubernetes.io/projected/128af9ad-0db4-46f5-bee7-993eabdfc2f4-kube-api-access-586wx\") pod \"opendatahub-operator-controller-manager-745bd4f5cc-pxsh5\" (UID: \"128af9ad-0db4-46f5-bee7-993eabdfc2f4\") " pod="opendatahub/opendatahub-operator-controller-manager-745bd4f5cc-pxsh5"
Jul 26 15:55:01.212877 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:01.212813    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/128af9ad-0db4-46f5-bee7-993eabdfc2f4-tmp\") pod \"opendatahub-operator-controller-manager-745bd4f5cc-pxsh5\" (UID: \"128af9ad-0db4-46f5-bee7-993eabdfc2f4\") " pod="opendatahub/opendatahub-operator-controller-manager-745bd4f5cc-pxsh5"
Jul 26 15:55:01.212877 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:01.212858    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/128af9ad-0db4-46f5-bee7-993eabdfc2f4-webhook-cert\") pod \"opendatahub-operator-controller-manager-745bd4f5cc-pxsh5\" (UID: \"128af9ad-0db4-46f5-bee7-993eabdfc2f4\") " pod="opendatahub/opendatahub-operator-controller-manager-745bd4f5cc-pxsh5"
Jul 26 15:55:01.212986 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:01.212928    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/128af9ad-0db4-46f5-bee7-993eabdfc2f4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-745bd4f5cc-pxsh5\" (UID: \"128af9ad-0db4-46f5-bee7-993eabdfc2f4\") " pod="opendatahub/opendatahub-operator-controller-manager-745bd4f5cc-pxsh5"
Jul 26 15:55:01.246575 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:01.246504    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:55:01.313757 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:01.313724    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/128af9ad-0db4-46f5-bee7-993eabdfc2f4-manifests\") pod \"opendatahub-operator-controller-manager-745bd4f5cc-pxsh5\" (UID: \"128af9ad-0db4-46f5-bee7-993eabdfc2f4\") " pod="opendatahub/opendatahub-operator-controller-manager-745bd4f5cc-pxsh5"
Jul 26 15:55:01.313931 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:01.313806    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-586wx\" (UniqueName: \"kubernetes.io/projected/128af9ad-0db4-46f5-bee7-993eabdfc2f4-kube-api-access-586wx\") pod \"opendatahub-operator-controller-manager-745bd4f5cc-pxsh5\" (UID: \"128af9ad-0db4-46f5-bee7-993eabdfc2f4\") " pod="opendatahub/opendatahub-operator-controller-manager-745bd4f5cc-pxsh5"
Jul 26 15:55:01.313931 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:01.313841    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/128af9ad-0db4-46f5-bee7-993eabdfc2f4-tmp\") pod \"opendatahub-operator-controller-manager-745bd4f5cc-pxsh5\" (UID: \"128af9ad-0db4-46f5-bee7-993eabdfc2f4\") " pod="opendatahub/opendatahub-operator-controller-manager-745bd4f5cc-pxsh5"
Jul 26 15:55:01.313931 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:01.313874    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/128af9ad-0db4-46f5-bee7-993eabdfc2f4-webhook-cert\") pod \"opendatahub-operator-controller-manager-745bd4f5cc-pxsh5\" (UID: \"128af9ad-0db4-46f5-bee7-993eabdfc2f4\") " pod="opendatahub/opendatahub-operator-controller-manager-745bd4f5cc-pxsh5"
Jul 26 15:55:01.313931 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:01.313903    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/128af9ad-0db4-46f5-bee7-993eabdfc2f4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-745bd4f5cc-pxsh5\" (UID: \"128af9ad-0db4-46f5-bee7-993eabdfc2f4\") " pod="opendatahub/opendatahub-operator-controller-manager-745bd4f5cc-pxsh5"
Jul 26 15:55:01.314221 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:01.314202    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/128af9ad-0db4-46f5-bee7-993eabdfc2f4-manifests\") pod \"opendatahub-operator-controller-manager-745bd4f5cc-pxsh5\" (UID: \"128af9ad-0db4-46f5-bee7-993eabdfc2f4\") " pod="opendatahub/opendatahub-operator-controller-manager-745bd4f5cc-pxsh5"
Jul 26 15:55:01.315940 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:01.315913    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/128af9ad-0db4-46f5-bee7-993eabdfc2f4-tmp\") pod \"opendatahub-operator-controller-manager-745bd4f5cc-pxsh5\" (UID: \"128af9ad-0db4-46f5-bee7-993eabdfc2f4\") " pod="opendatahub/opendatahub-operator-controller-manager-745bd4f5cc-pxsh5"
Jul 26 15:55:01.316282 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:01.316264    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/128af9ad-0db4-46f5-bee7-993eabdfc2f4-webhook-cert\") pod \"opendatahub-operator-controller-manager-745bd4f5cc-pxsh5\" (UID: \"128af9ad-0db4-46f5-bee7-993eabdfc2f4\") " pod="opendatahub/opendatahub-operator-controller-manager-745bd4f5cc-pxsh5"
Jul 26 15:55:01.316352 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:01.316308    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/128af9ad-0db4-46f5-bee7-993eabdfc2f4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-745bd4f5cc-pxsh5\" (UID: \"128af9ad-0db4-46f5-bee7-993eabdfc2f4\") " pod="opendatahub/opendatahub-operator-controller-manager-745bd4f5cc-pxsh5"
Jul 26 15:55:01.321863 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:01.321838    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-586wx\" (UniqueName: \"kubernetes.io/projected/128af9ad-0db4-46f5-bee7-993eabdfc2f4-kube-api-access-586wx\") pod \"opendatahub-operator-controller-manager-745bd4f5cc-pxsh5\" (UID: \"128af9ad-0db4-46f5-bee7-993eabdfc2f4\") " pod="opendatahub/opendatahub-operator-controller-manager-745bd4f5cc-pxsh5"
Jul 26 15:55:01.353204 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:01.353159    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:55:01.445901 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:01.445866    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:55:01.450080 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:01.450061    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-745bd4f5cc-pxsh5"
Jul 26 15:55:01.546595 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:01.546553    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:55:01.584011 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:01.583989    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-745bd4f5cc-pxsh5"]
Jul 26 15:55:01.586705 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:55:01.586677    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod128af9ad_0db4_46f5_bee7_993eabdfc2f4.slice/crio-a6cef6826a9405d04211be814fcbe9ed57c6da499fcd875258d0f61eb8984cbf WatchSource:0}: Error finding container a6cef6826a9405d04211be814fcbe9ed57c6da499fcd875258d0f61eb8984cbf: Status 404 returned error can't find the container with id a6cef6826a9405d04211be814fcbe9ed57c6da499fcd875258d0f61eb8984cbf
Jul 26 15:55:01.646590 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:01.646549    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:55:01.719552 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:01.719517    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-745bd4f5cc-pxsh5" event={"ID":"128af9ad-0db4-46f5-bee7-993eabdfc2f4","Type":"ContainerStarted","Data":"a6cef6826a9405d04211be814fcbe9ed57c6da499fcd875258d0f61eb8984cbf"}
Jul 26 15:55:01.746357 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:01.746311    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:55:04.733961 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:04.733926    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-745bd4f5cc-pxsh5" event={"ID":"128af9ad-0db4-46f5-bee7-993eabdfc2f4","Type":"ContainerStarted","Data":"d9309d773387f8c8a3c9591ad730b1cadde489004de7b7b5f9044d129a99767a"}
Jul 26 15:55:05.742345 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:05.742309    2563 generic.go:356] "Generic (PLEG): container finished" podID="128af9ad-0db4-46f5-bee7-993eabdfc2f4" containerID="d9309d773387f8c8a3c9591ad730b1cadde489004de7b7b5f9044d129a99767a" exitCode=0
Jul 26 15:55:05.742788 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:05.742392    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-745bd4f5cc-pxsh5" event={"ID":"128af9ad-0db4-46f5-bee7-993eabdfc2f4","Type":"ContainerDied","Data":"d9309d773387f8c8a3c9591ad730b1cadde489004de7b7b5f9044d129a99767a"}
Jul 26 15:55:06.747738 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:06.747701    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-745bd4f5cc-pxsh5" event={"ID":"128af9ad-0db4-46f5-bee7-993eabdfc2f4","Type":"ContainerStarted","Data":"a9bb7671f89cff8821c4c5c646abcca1600385f58bfdaf3ec58b6a44cf4c99fd"}
Jul 26 15:55:06.748134 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:06.747902    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-745bd4f5cc-pxsh5"
Jul 26 15:55:06.773665 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:06.773617    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-745bd4f5cc-pxsh5" podStartSLOduration=1.505110239 podStartE2EDuration="5.773600596s" totalImagesPullingTime="3.268307076s" totalInitContainerRuntime="1.000183281s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:55:01 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:55:06.772872797 +0000 UTC m=+298.877098617" watchObservedRunningTime="2026-07-26 15:55:06.773600596 +0000 UTC m=+298.877826388"
Jul 26 15:55:07.497846 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:07.497812    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:55:08.271345 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:08.271321    2563 kubelet.go:1648] "Image garbage collection succeeded"
Jul 26 15:55:08.752539 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:08.752493    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:55:11.521838 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:11.521800    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:55:11.643477 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:11.643446    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:55:12.865708 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:12.865653    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:55:12.973180 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:12.973142    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:55:17.754391 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:17.754362    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-745bd4f5cc-pxsh5"
Jul 26 15:55:20.280625 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:20.280578    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:55:20.397025 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:20.396979    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:55:20.955193 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:20.955162    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-lr2rv"]
Jul 26 15:55:20.964235 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:20.964212    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-lr2rv"
Jul 26 15:55:20.966489 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:20.966469    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-2kmcl\""
Jul 26 15:55:20.968089 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:20.968073    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Jul 26 15:55:20.975600 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:20.975579    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-lr2rv"]
Jul 26 15:55:21.016944 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:21.016916    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:55:21.090164 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:21.090126    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtk4c\" (UniqueName: \"kubernetes.io/projected/b23f9014-7117-43d6-975d-4f4e945db935-kube-api-access-gtk4c\") pod \"odh-model-controller-7998c67dc4-lr2rv\" (UID: \"b23f9014-7117-43d6-975d-4f4e945db935\") " pod="opendatahub/odh-model-controller-7998c67dc4-lr2rv"
Jul 26 15:55:21.090326 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:21.090198    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b23f9014-7117-43d6-975d-4f4e945db935-cert\") pod \"odh-model-controller-7998c67dc4-lr2rv\" (UID: \"b23f9014-7117-43d6-975d-4f4e945db935\") " pod="opendatahub/odh-model-controller-7998c67dc4-lr2rv"
Jul 26 15:55:21.119172 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:21.119141    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:55:21.191153 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:21.191112    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtk4c\" (UniqueName: \"kubernetes.io/projected/b23f9014-7117-43d6-975d-4f4e945db935-kube-api-access-gtk4c\") pod \"odh-model-controller-7998c67dc4-lr2rv\" (UID: \"b23f9014-7117-43d6-975d-4f4e945db935\") " pod="opendatahub/odh-model-controller-7998c67dc4-lr2rv"
Jul 26 15:55:21.191320 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:21.191180    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b23f9014-7117-43d6-975d-4f4e945db935-cert\") pod \"odh-model-controller-7998c67dc4-lr2rv\" (UID: \"b23f9014-7117-43d6-975d-4f4e945db935\") " pod="opendatahub/odh-model-controller-7998c67dc4-lr2rv"
Jul 26 15:55:21.193808 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:21.193753    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b23f9014-7117-43d6-975d-4f4e945db935-cert\") pod \"odh-model-controller-7998c67dc4-lr2rv\" (UID: \"b23f9014-7117-43d6-975d-4f4e945db935\") " pod="opendatahub/odh-model-controller-7998c67dc4-lr2rv"
Jul 26 15:55:21.199418 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:21.199397    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtk4c\" (UniqueName: \"kubernetes.io/projected/b23f9014-7117-43d6-975d-4f4e945db935-kube-api-access-gtk4c\") pod \"odh-model-controller-7998c67dc4-lr2rv\" (UID: \"b23f9014-7117-43d6-975d-4f4e945db935\") " pod="opendatahub/odh-model-controller-7998c67dc4-lr2rv"
Jul 26 15:55:21.211342 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:21.211284    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:55:21.273936 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:21.273907    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-lr2rv"
Jul 26 15:55:21.400695 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:21.400665    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-lr2rv"]
Jul 26 15:55:21.404086 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:55:21.404062    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb23f9014_7117_43d6_975d_4f4e945db935.slice/crio-92c08c5de014154d3de022209fa2aed6f7252700b31f4af8e310e8ad6dc3a02a WatchSource:0}: Error finding container 92c08c5de014154d3de022209fa2aed6f7252700b31f4af8e310e8ad6dc3a02a: Status 404 returned error can't find the container with id 92c08c5de014154d3de022209fa2aed6f7252700b31f4af8e310e8ad6dc3a02a
Jul 26 15:55:21.405512 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:21.405495    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 26 15:55:21.587319 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:21.587290    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:55:21.616350 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:21.616312    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-jdqzl"]
Jul 26 15:55:21.619691 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:21.619673    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-jdqzl"
Jul 26 15:55:21.622845 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:21.622820    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Jul 26 15:55:21.623005 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:21.622973    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-5k5bp\""
Jul 26 15:55:21.630654 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:21.630628    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-jdqzl"]
Jul 26 15:55:21.753729 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:21.753695    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:55:21.796613 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:21.796558    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrgjs\" (UniqueName: \"kubernetes.io/projected/70f22506-ae20-4e4b-873f-0bf16997cff6-kube-api-access-jrgjs\") pod \"model-serving-api-54d89bd5fc-jdqzl\" (UID: \"70f22506-ae20-4e4b-873f-0bf16997cff6\") " pod="opendatahub/model-serving-api-54d89bd5fc-jdqzl"
Jul 26 15:55:21.796783 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:21.796660    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70f22506-ae20-4e4b-873f-0bf16997cff6-tls-certs\") pod \"model-serving-api-54d89bd5fc-jdqzl\" (UID: \"70f22506-ae20-4e4b-873f-0bf16997cff6\") " pod="opendatahub/model-serving-api-54d89bd5fc-jdqzl"
Jul 26 15:55:21.803890 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:21.803843    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-lr2rv" event={"ID":"b23f9014-7117-43d6-975d-4f4e945db935","Type":"ContainerStarted","Data":"92c08c5de014154d3de022209fa2aed6f7252700b31f4af8e310e8ad6dc3a02a"}
Jul 26 15:55:21.897509 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:21.897416    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70f22506-ae20-4e4b-873f-0bf16997cff6-tls-certs\") pod \"model-serving-api-54d89bd5fc-jdqzl\" (UID: \"70f22506-ae20-4e4b-873f-0bf16997cff6\") " pod="opendatahub/model-serving-api-54d89bd5fc-jdqzl"
Jul 26 15:55:21.897509 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:21.897504    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrgjs\" (UniqueName: \"kubernetes.io/projected/70f22506-ae20-4e4b-873f-0bf16997cff6-kube-api-access-jrgjs\") pod \"model-serving-api-54d89bd5fc-jdqzl\" (UID: \"70f22506-ae20-4e4b-873f-0bf16997cff6\") " pod="opendatahub/model-serving-api-54d89bd5fc-jdqzl"
Jul 26 15:55:21.899916 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:21.899894    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70f22506-ae20-4e4b-873f-0bf16997cff6-tls-certs\") pod \"model-serving-api-54d89bd5fc-jdqzl\" (UID: \"70f22506-ae20-4e4b-873f-0bf16997cff6\") " pod="opendatahub/model-serving-api-54d89bd5fc-jdqzl"
Jul 26 15:55:21.905737 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:21.905713    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrgjs\" (UniqueName: \"kubernetes.io/projected/70f22506-ae20-4e4b-873f-0bf16997cff6-kube-api-access-jrgjs\") pod \"model-serving-api-54d89bd5fc-jdqzl\" (UID: \"70f22506-ae20-4e4b-873f-0bf16997cff6\") " pod="opendatahub/model-serving-api-54d89bd5fc-jdqzl"
Jul 26 15:55:21.931676 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:21.931648    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-jdqzl"
Jul 26 15:55:22.075524 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:22.075496    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-jdqzl"]
Jul 26 15:55:22.078406 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:55:22.078368    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70f22506_ae20_4e4b_873f_0bf16997cff6.slice/crio-5c9fc8cfb18389ec3d86263ed04cdd19e43f8f9456a8cd7c2e395207ac56f1f1 WatchSource:0}: Error finding container 5c9fc8cfb18389ec3d86263ed04cdd19e43f8f9456a8cd7c2e395207ac56f1f1: Status 404 returned error can't find the container with id 5c9fc8cfb18389ec3d86263ed04cdd19e43f8f9456a8cd7c2e395207ac56f1f1
Jul 26 15:55:22.810194 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:22.810153    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-jdqzl" event={"ID":"70f22506-ae20-4e4b-873f-0bf16997cff6","Type":"ContainerStarted","Data":"5c9fc8cfb18389ec3d86263ed04cdd19e43f8f9456a8cd7c2e395207ac56f1f1"}
Jul 26 15:55:24.821199 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:24.821161    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-lr2rv" event={"ID":"b23f9014-7117-43d6-975d-4f4e945db935","Type":"ContainerStarted","Data":"99b2568c9bfe5aa26e4a651af49e8e710a419ecfe30c1863e40ea7c653c9cdd0"}
Jul 26 15:55:24.821674 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:24.821278    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-lr2rv"
Jul 26 15:55:24.822931 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:24.822906    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-jdqzl" event={"ID":"70f22506-ae20-4e4b-873f-0bf16997cff6","Type":"ContainerStarted","Data":"1eeac66cca9d26d79080d6314766191f3842b5dce0be2a0d63bc9a3caa86c33b"}
Jul 26 15:55:24.823061 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:24.823036    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-jdqzl"
Jul 26 15:55:24.836321 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:24.835791    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-lr2rv" podStartSLOduration=1.578490413 podStartE2EDuration="4.835752518s" totalImagesPullingTime="3.257262105s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:55:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:55:24.83503642 +0000 UTC m=+316.939262228" watchObservedRunningTime="2026-07-26 15:55:24.835752518 +0000 UTC m=+316.939978326"
Jul 26 15:55:24.854088 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:24.854033    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-jdqzl" podStartSLOduration=1.2751749829999999 podStartE2EDuration="3.854015364s" totalImagesPullingTime="2.578840381s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:55:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:55:24.852565402 +0000 UTC m=+316.956791212" watchObservedRunningTime="2026-07-26 15:55:24.854015364 +0000 UTC m=+316.958241170"
Jul 26 15:55:30.724390 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:30.724347    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:55:30.775815 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:30.775785    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-7p94c"]
Jul 26 15:55:30.779021 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:30.778998    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-7p94c"
Jul 26 15:55:30.781203 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:30.781177    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Jul 26 15:55:30.781383 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:30.781361    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-t2lr5\""
Jul 26 15:55:30.784945 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:30.784922    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-t5q6d"]
Jul 26 15:55:30.785075 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:30.785018    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9e359ea2-4eda-44ed-a0fc-f7885f2891ab-cert\") pod \"kserve-controller-manager-d6f4b65b8-7p94c\" (UID: \"9e359ea2-4eda-44ed-a0fc-f7885f2891ab\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-7p94c"
Jul 26 15:55:30.785144 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:30.785092    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpcmt\" (UniqueName: \"kubernetes.io/projected/9e359ea2-4eda-44ed-a0fc-f7885f2891ab-kube-api-access-lpcmt\") pod \"kserve-controller-manager-d6f4b65b8-7p94c\" (UID: \"9e359ea2-4eda-44ed-a0fc-f7885f2891ab\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-7p94c"
Jul 26 15:55:30.788640 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:30.788620    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t5q6d"
Jul 26 15:55:30.789902 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:30.789851    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-7p94c"]
Jul 26 15:55:30.790937 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:30.790920    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-c9w6h\""
Jul 26 15:55:30.792330 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:30.792313    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Jul 26 15:55:30.797338 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:30.797128    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-t5q6d"]
Jul 26 15:55:30.822420 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:30.822389    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:55:30.885975 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:30.885941    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jrwg\" (UniqueName: \"kubernetes.io/projected/26ad813d-5c13-4977-ad9e-09342b76426e-kube-api-access-4jrwg\") pod \"llmisvc-controller-manager-777c6bd7d5-t5q6d\" (UID: \"26ad813d-5c13-4977-ad9e-09342b76426e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t5q6d"
Jul 26 15:55:30.886182 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:30.885991    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpcmt\" (UniqueName: \"kubernetes.io/projected/9e359ea2-4eda-44ed-a0fc-f7885f2891ab-kube-api-access-lpcmt\") pod \"kserve-controller-manager-d6f4b65b8-7p94c\" (UID: \"9e359ea2-4eda-44ed-a0fc-f7885f2891ab\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-7p94c"
Jul 26 15:55:30.886182 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:30.886082    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/26ad813d-5c13-4977-ad9e-09342b76426e-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-t5q6d\" (UID: \"26ad813d-5c13-4977-ad9e-09342b76426e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t5q6d"
Jul 26 15:55:30.886182 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:30.886111    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9e359ea2-4eda-44ed-a0fc-f7885f2891ab-cert\") pod \"kserve-controller-manager-d6f4b65b8-7p94c\" (UID: \"9e359ea2-4eda-44ed-a0fc-f7885f2891ab\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-7p94c"
Jul 26 15:55:30.886334 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:55:30.886215    2563 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Jul 26 15:55:30.886334 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:55:30.886276    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e359ea2-4eda-44ed-a0fc-f7885f2891ab-cert podName:9e359ea2-4eda-44ed-a0fc-f7885f2891ab nodeName:}" failed. No retries permitted until 2026-07-26 15:55:31.386256832 +0000 UTC m=+323.490482630 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/9e359ea2-4eda-44ed-a0fc-f7885f2891ab-cert") pod "kserve-controller-manager-d6f4b65b8-7p94c" (UID: "9e359ea2-4eda-44ed-a0fc-f7885f2891ab") : secret "kserve-webhook-server-cert" not found
Jul 26 15:55:30.898477 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:30.898441    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpcmt\" (UniqueName: \"kubernetes.io/projected/9e359ea2-4eda-44ed-a0fc-f7885f2891ab-kube-api-access-lpcmt\") pod \"kserve-controller-manager-d6f4b65b8-7p94c\" (UID: \"9e359ea2-4eda-44ed-a0fc-f7885f2891ab\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-7p94c"
Jul 26 15:55:30.986938 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:30.986842    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jrwg\" (UniqueName: \"kubernetes.io/projected/26ad813d-5c13-4977-ad9e-09342b76426e-kube-api-access-4jrwg\") pod \"llmisvc-controller-manager-777c6bd7d5-t5q6d\" (UID: \"26ad813d-5c13-4977-ad9e-09342b76426e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t5q6d"
Jul 26 15:55:30.986938 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:30.986938    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/26ad813d-5c13-4977-ad9e-09342b76426e-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-t5q6d\" (UID: \"26ad813d-5c13-4977-ad9e-09342b76426e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t5q6d"
Jul 26 15:55:30.987171 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:55:30.987012    2563 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Jul 26 15:55:30.987171 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:55:30.987061    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26ad813d-5c13-4977-ad9e-09342b76426e-cert podName:26ad813d-5c13-4977-ad9e-09342b76426e nodeName:}" failed. No retries permitted until 2026-07-26 15:55:31.487046167 +0000 UTC m=+323.591271953 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/26ad813d-5c13-4977-ad9e-09342b76426e-cert") pod "llmisvc-controller-manager-777c6bd7d5-t5q6d" (UID: "26ad813d-5c13-4977-ad9e-09342b76426e") : secret "llmisvc-webhook-server-cert" not found
Jul 26 15:55:30.999152 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:30.999128    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jrwg\" (UniqueName: \"kubernetes.io/projected/26ad813d-5c13-4977-ad9e-09342b76426e-kube-api-access-4jrwg\") pod \"llmisvc-controller-manager-777c6bd7d5-t5q6d\" (UID: \"26ad813d-5c13-4977-ad9e-09342b76426e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t5q6d"
Jul 26 15:55:31.045473 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:31.045433    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:55:31.156094 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:31.156064    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:55:31.250152 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:31.250073    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:55:31.268488 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:31.268456    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5bfd866646-9w5w8"]
Jul 26 15:55:31.272113 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:31.272092    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5bfd866646-9w5w8"
Jul 26 15:55:31.275382 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:31.275359    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 26 15:55:31.275575 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:31.275530    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 26 15:55:31.275704 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:31.275601    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-cgfpv\""
Jul 26 15:55:31.275704 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:31.275688    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 26 15:55:31.275862 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:31.275811    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 26 15:55:31.282135 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:31.282105    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5bfd866646-9w5w8"]
Jul 26 15:55:31.289368 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:31.289349    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc2gz\" (UniqueName: \"kubernetes.io/projected/c68b8fda-d0c9-479a-a3ed-dbef655aa0f7-kube-api-access-vc2gz\") pod \"kube-auth-proxy-5bfd866646-9w5w8\" (UID: \"c68b8fda-d0c9-479a-a3ed-dbef655aa0f7\") " pod="openshift-ingress/kube-auth-proxy-5bfd866646-9w5w8"
Jul 26 15:55:31.289457 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:31.289379    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c68b8fda-d0c9-479a-a3ed-dbef655aa0f7-tls-certs\") pod \"kube-auth-proxy-5bfd866646-9w5w8\" (UID: \"c68b8fda-d0c9-479a-a3ed-dbef655aa0f7\") " pod="openshift-ingress/kube-auth-proxy-5bfd866646-9w5w8"
Jul 26 15:55:31.289457 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:31.289439    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c68b8fda-d0c9-479a-a3ed-dbef655aa0f7-tmp\") pod \"kube-auth-proxy-5bfd866646-9w5w8\" (UID: \"c68b8fda-d0c9-479a-a3ed-dbef655aa0f7\") " pod="openshift-ingress/kube-auth-proxy-5bfd866646-9w5w8"
Jul 26 15:55:31.390358 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:31.390321    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc2gz\" (UniqueName: \"kubernetes.io/projected/c68b8fda-d0c9-479a-a3ed-dbef655aa0f7-kube-api-access-vc2gz\") pod \"kube-auth-proxy-5bfd866646-9w5w8\" (UID: \"c68b8fda-d0c9-479a-a3ed-dbef655aa0f7\") " pod="openshift-ingress/kube-auth-proxy-5bfd866646-9w5w8"
Jul 26 15:55:31.390567 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:31.390367    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c68b8fda-d0c9-479a-a3ed-dbef655aa0f7-tls-certs\") pod \"kube-auth-proxy-5bfd866646-9w5w8\" (UID: \"c68b8fda-d0c9-479a-a3ed-dbef655aa0f7\") " pod="openshift-ingress/kube-auth-proxy-5bfd866646-9w5w8"
Jul 26 15:55:31.390567 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:31.390428    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c68b8fda-d0c9-479a-a3ed-dbef655aa0f7-tmp\") pod \"kube-auth-proxy-5bfd866646-9w5w8\" (UID: \"c68b8fda-d0c9-479a-a3ed-dbef655aa0f7\") " pod="openshift-ingress/kube-auth-proxy-5bfd866646-9w5w8"
Jul 26 15:55:31.390567 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:31.390447    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9e359ea2-4eda-44ed-a0fc-f7885f2891ab-cert\") pod \"kserve-controller-manager-d6f4b65b8-7p94c\" (UID: \"9e359ea2-4eda-44ed-a0fc-f7885f2891ab\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-7p94c"
Jul 26 15:55:31.390567 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:55:31.390508    2563 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Jul 26 15:55:31.390801 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:55:31.390579    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c68b8fda-d0c9-479a-a3ed-dbef655aa0f7-tls-certs podName:c68b8fda-d0c9-479a-a3ed-dbef655aa0f7 nodeName:}" failed. No retries permitted until 2026-07-26 15:55:31.8905589 +0000 UTC m=+323.994784689 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/c68b8fda-d0c9-479a-a3ed-dbef655aa0f7-tls-certs") pod "kube-auth-proxy-5bfd866646-9w5w8" (UID: "c68b8fda-d0c9-479a-a3ed-dbef655aa0f7") : secret "kube-auth-proxy-tls" not found
Jul 26 15:55:31.392736 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:31.392706    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c68b8fda-d0c9-479a-a3ed-dbef655aa0f7-tmp\") pod \"kube-auth-proxy-5bfd866646-9w5w8\" (UID: \"c68b8fda-d0c9-479a-a3ed-dbef655aa0f7\") " pod="openshift-ingress/kube-auth-proxy-5bfd866646-9w5w8"
Jul 26 15:55:31.393012 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:31.392993    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9e359ea2-4eda-44ed-a0fc-f7885f2891ab-cert\") pod \"kserve-controller-manager-d6f4b65b8-7p94c\" (UID: \"9e359ea2-4eda-44ed-a0fc-f7885f2891ab\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-7p94c"
Jul 26 15:55:31.402503 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:31.402479    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc2gz\" (UniqueName: \"kubernetes.io/projected/c68b8fda-d0c9-479a-a3ed-dbef655aa0f7-kube-api-access-vc2gz\") pod \"kube-auth-proxy-5bfd866646-9w5w8\" (UID: \"c68b8fda-d0c9-479a-a3ed-dbef655aa0f7\") " pod="openshift-ingress/kube-auth-proxy-5bfd866646-9w5w8"
Jul 26 15:55:31.491308 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:31.491273    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/26ad813d-5c13-4977-ad9e-09342b76426e-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-t5q6d\" (UID: \"26ad813d-5c13-4977-ad9e-09342b76426e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t5q6d"
Jul 26 15:55:31.493644 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:31.493621    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/26ad813d-5c13-4977-ad9e-09342b76426e-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-t5q6d\" (UID: \"26ad813d-5c13-4977-ad9e-09342b76426e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t5q6d"
Jul 26 15:55:31.524863 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:31.524750    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:55:31.691061 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:31.691030    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-7p94c"
Jul 26 15:55:31.702233 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:31.702210    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t5q6d"
Jul 26 15:55:31.888949 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:55:31.888917    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e359ea2_4eda_44ed_a0fc_f7885f2891ab.slice/crio-ac0b733284901db751c58139ac2d35da32328070d3dafa9f21bdc6e12dbb3186 WatchSource:0}: Error finding container ac0b733284901db751c58139ac2d35da32328070d3dafa9f21bdc6e12dbb3186: Status 404 returned error can't find the container with id ac0b733284901db751c58139ac2d35da32328070d3dafa9f21bdc6e12dbb3186
Jul 26 15:55:31.889342 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:31.889234    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-7p94c"]
Jul 26 15:55:31.898086 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:31.898063    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c68b8fda-d0c9-479a-a3ed-dbef655aa0f7-tls-certs\") pod \"kube-auth-proxy-5bfd866646-9w5w8\" (UID: \"c68b8fda-d0c9-479a-a3ed-dbef655aa0f7\") " pod="openshift-ingress/kube-auth-proxy-5bfd866646-9w5w8"
Jul 26 15:55:31.900305 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:31.900283    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c68b8fda-d0c9-479a-a3ed-dbef655aa0f7-tls-certs\") pod \"kube-auth-proxy-5bfd866646-9w5w8\" (UID: \"c68b8fda-d0c9-479a-a3ed-dbef655aa0f7\") " pod="openshift-ingress/kube-auth-proxy-5bfd866646-9w5w8"
Jul 26 15:55:31.907429 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:31.907404    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-t5q6d"]
Jul 26 15:55:31.907863 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:55:31.907839    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod26ad813d_5c13_4977_ad9e_09342b76426e.slice/crio-2d77a5051bc5865c613414675d2d92a7bf0b39669ea96e3d17722f809b396162 WatchSource:0}: Error finding container 2d77a5051bc5865c613414675d2d92a7bf0b39669ea96e3d17722f809b396162: Status 404 returned error can't find the container with id 2d77a5051bc5865c613414675d2d92a7bf0b39669ea96e3d17722f809b396162
Jul 26 15:55:32.038519 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:32.038485    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:55:32.137983 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:32.137892    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:55:32.182889 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:32.182857    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5bfd866646-9w5w8"
Jul 26 15:55:32.238927 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:32.238895    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:55:32.317742 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:32.317715    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5bfd866646-9w5w8"]
Jul 26 15:55:32.320024 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:55:32.319997    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc68b8fda_d0c9_479a_a3ed_dbef655aa0f7.slice/crio-c8ef94f09c57a79e017a02a0da241afee8810c8b30403a75ce31c683be74dca2 WatchSource:0}: Error finding container c8ef94f09c57a79e017a02a0da241afee8810c8b30403a75ce31c683be74dca2: Status 404 returned error can't find the container with id c8ef94f09c57a79e017a02a0da241afee8810c8b30403a75ce31c683be74dca2
Jul 26 15:55:32.861284 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:32.861243    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5bfd866646-9w5w8" event={"ID":"c68b8fda-d0c9-479a-a3ed-dbef655aa0f7","Type":"ContainerStarted","Data":"c8ef94f09c57a79e017a02a0da241afee8810c8b30403a75ce31c683be74dca2"}
Jul 26 15:55:32.865022 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:32.864057    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-7p94c" event={"ID":"9e359ea2-4eda-44ed-a0fc-f7885f2891ab","Type":"ContainerStarted","Data":"ac0b733284901db751c58139ac2d35da32328070d3dafa9f21bdc6e12dbb3186"}
Jul 26 15:55:32.870792 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:32.870743    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t5q6d" event={"ID":"26ad813d-5c13-4977-ad9e-09342b76426e","Type":"ContainerStarted","Data":"2d77a5051bc5865c613414675d2d92a7bf0b39669ea96e3d17722f809b396162"}
Jul 26 15:55:33.063303 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:33.063262    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:55:33.071862 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:33.071677    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-5mp5d"]
Jul 26 15:55:33.076296 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:33.076269    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5mp5d"
Jul 26 15:55:33.078891 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:33.078714    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-ghtnr\""
Jul 26 15:55:33.079251 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:33.079033    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 26 15:55:33.079251 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:33.079036    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 26 15:55:33.079251 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:33.079125    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 26 15:55:33.079251 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:33.079036    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 26 15:55:33.090094 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:33.090065    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-5mp5d"]
Jul 26 15:55:33.109363 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:33.109312    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4ng8\" (UniqueName: \"kubernetes.io/projected/9d5f0889-c6a8-4641-b6f4-28e81a9a7955-kube-api-access-d4ng8\") pod \"istiod-openshift-gateway-5677ccb49b-5mp5d\" (UID: \"9d5f0889-c6a8-4641-b6f4-28e81a9a7955\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5mp5d"
Jul 26 15:55:33.109668 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:33.109368    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d5f0889-c6a8-4641-b6f4-28e81a9a7955-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-5mp5d\" (UID: \"9d5f0889-c6a8-4641-b6f4-28e81a9a7955\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5mp5d"
Jul 26 15:55:33.109668 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:33.109428    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9d5f0889-c6a8-4641-b6f4-28e81a9a7955-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-5mp5d\" (UID: \"9d5f0889-c6a8-4641-b6f4-28e81a9a7955\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5mp5d"
Jul 26 15:55:33.109668 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:33.109471    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d5f0889-c6a8-4641-b6f4-28e81a9a7955-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-5mp5d\" (UID: \"9d5f0889-c6a8-4641-b6f4-28e81a9a7955\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5mp5d"
Jul 26 15:55:33.109668 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:33.109549    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9d5f0889-c6a8-4641-b6f4-28e81a9a7955-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-5mp5d\" (UID: \"9d5f0889-c6a8-4641-b6f4-28e81a9a7955\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5mp5d"
Jul 26 15:55:33.109668 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:33.109593    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9d5f0889-c6a8-4641-b6f4-28e81a9a7955-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-5mp5d\" (UID: \"9d5f0889-c6a8-4641-b6f4-28e81a9a7955\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5mp5d"
Jul 26 15:55:33.109668 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:33.109647    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9d5f0889-c6a8-4641-b6f4-28e81a9a7955-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-5mp5d\" (UID: \"9d5f0889-c6a8-4641-b6f4-28e81a9a7955\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5mp5d"
Jul 26 15:55:33.159666 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:33.159569    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:55:33.211004 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:33.210960    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9d5f0889-c6a8-4641-b6f4-28e81a9a7955-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-5mp5d\" (UID: \"9d5f0889-c6a8-4641-b6f4-28e81a9a7955\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5mp5d"
Jul 26 15:55:33.211181 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:33.211031    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9d5f0889-c6a8-4641-b6f4-28e81a9a7955-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-5mp5d\" (UID: \"9d5f0889-c6a8-4641-b6f4-28e81a9a7955\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5mp5d"
Jul 26 15:55:33.211181 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:33.211090    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4ng8\" (UniqueName: \"kubernetes.io/projected/9d5f0889-c6a8-4641-b6f4-28e81a9a7955-kube-api-access-d4ng8\") pod \"istiod-openshift-gateway-5677ccb49b-5mp5d\" (UID: \"9d5f0889-c6a8-4641-b6f4-28e81a9a7955\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5mp5d"
Jul 26 15:55:33.211181 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:33.211122    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d5f0889-c6a8-4641-b6f4-28e81a9a7955-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-5mp5d\" (UID: \"9d5f0889-c6a8-4641-b6f4-28e81a9a7955\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5mp5d"
Jul 26 15:55:33.211181 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:33.211159    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9d5f0889-c6a8-4641-b6f4-28e81a9a7955-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-5mp5d\" (UID: \"9d5f0889-c6a8-4641-b6f4-28e81a9a7955\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5mp5d"
Jul 26 15:55:33.211392 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:33.211196    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d5f0889-c6a8-4641-b6f4-28e81a9a7955-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-5mp5d\" (UID: \"9d5f0889-c6a8-4641-b6f4-28e81a9a7955\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5mp5d"
Jul 26 15:55:33.211392 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:33.211241    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9d5f0889-c6a8-4641-b6f4-28e81a9a7955-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-5mp5d\" (UID: \"9d5f0889-c6a8-4641-b6f4-28e81a9a7955\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5mp5d"
Jul 26 15:55:33.212011 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:33.211960    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9d5f0889-c6a8-4641-b6f4-28e81a9a7955-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-5mp5d\" (UID: \"9d5f0889-c6a8-4641-b6f4-28e81a9a7955\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5mp5d"
Jul 26 15:55:33.214241 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:33.214166    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9d5f0889-c6a8-4641-b6f4-28e81a9a7955-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-5mp5d\" (UID: \"9d5f0889-c6a8-4641-b6f4-28e81a9a7955\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5mp5d"
Jul 26 15:55:33.215010 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:33.214977    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d5f0889-c6a8-4641-b6f4-28e81a9a7955-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-5mp5d\" (UID: \"9d5f0889-c6a8-4641-b6f4-28e81a9a7955\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5mp5d"
Jul 26 15:55:33.215388 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:33.215360    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9d5f0889-c6a8-4641-b6f4-28e81a9a7955-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-5mp5d\" (UID: \"9d5f0889-c6a8-4641-b6f4-28e81a9a7955\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5mp5d"
Jul 26 15:55:33.217823 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:33.217740    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9d5f0889-c6a8-4641-b6f4-28e81a9a7955-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-5mp5d\" (UID: \"9d5f0889-c6a8-4641-b6f4-28e81a9a7955\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5mp5d"
Jul 26 15:55:33.219745 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:33.219703    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d5f0889-c6a8-4641-b6f4-28e81a9a7955-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-5mp5d\" (UID: \"9d5f0889-c6a8-4641-b6f4-28e81a9a7955\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5mp5d"
Jul 26 15:55:33.221555 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:33.221531    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4ng8\" (UniqueName: \"kubernetes.io/projected/9d5f0889-c6a8-4641-b6f4-28e81a9a7955-kube-api-access-d4ng8\") pod \"istiod-openshift-gateway-5677ccb49b-5mp5d\" (UID: \"9d5f0889-c6a8-4641-b6f4-28e81a9a7955\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5mp5d"
Jul 26 15:55:33.260973 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:33.260885    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:55:33.391249 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:33.391200    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5mp5d"
Jul 26 15:55:33.625868 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:33.625825    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-5mp5d"]
Jul 26 15:55:34.207418 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:55:34.207384    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d5f0889_c6a8_4641_b6f4_28e81a9a7955.slice/crio-0d1a91ee7bf746bb3ce54f6ffe3b7e891fe84550a0945d8d21d14047b748968e WatchSource:0}: Error finding container 0d1a91ee7bf746bb3ce54f6ffe3b7e891fe84550a0945d8d21d14047b748968e: Status 404 returned error can't find the container with id 0d1a91ee7bf746bb3ce54f6ffe3b7e891fe84550a0945d8d21d14047b748968e
Jul 26 15:55:34.468165 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:34.468083    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:55:34.882463 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:34.882426    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5mp5d" event={"ID":"9d5f0889-c6a8-4641-b6f4-28e81a9a7955","Type":"ContainerStarted","Data":"0d1a91ee7bf746bb3ce54f6ffe3b7e891fe84550a0945d8d21d14047b748968e"}
Jul 26 15:55:35.829791 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:35.829742    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-lr2rv"
Jul 26 15:55:35.832077 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:35.832056    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-jdqzl"
Jul 26 15:55:35.896022 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:35.895586    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-7p94c" event={"ID":"9e359ea2-4eda-44ed-a0fc-f7885f2891ab","Type":"ContainerStarted","Data":"ff4fc38a69a23d4d1da250911a95eb8b5bd8e479da95034215149d40aaf6af55"}
Jul 26 15:55:35.896207 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:35.896044    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-7p94c"
Jul 26 15:55:35.897871 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:35.897801    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t5q6d" event={"ID":"26ad813d-5c13-4977-ad9e-09342b76426e","Type":"ContainerStarted","Data":"bba7f0e0e8d22373f9877b7dc57f2893334798b4e540f2ae8642799f2b33c8e8"}
Jul 26 15:55:35.898001 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:35.897900    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t5q6d"
Jul 26 15:55:35.899591 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:35.899566    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5bfd866646-9w5w8" event={"ID":"c68b8fda-d0c9-479a-a3ed-dbef655aa0f7","Type":"ContainerStarted","Data":"76ddbad2732990c28c931a26e189b8b54263f443d660fcb6325c351656f04421"}
Jul 26 15:55:35.915134 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:35.915080    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-7p94c" podStartSLOduration=2.340364391 podStartE2EDuration="5.915062191s" totalImagesPullingTime="3.5746978s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:55:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:55:35.9136883 +0000 UTC m=+328.017914108" watchObservedRunningTime="2026-07-26 15:55:35.915062191 +0000 UTC m=+328.019287999"
Jul 26 15:55:35.930970 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:35.930910    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5bfd866646-9w5w8" podStartSLOduration=1.7852438400000001 podStartE2EDuration="4.93089024s" totalImagesPullingTime="3.1456464s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:55:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:55:35.929480871 +0000 UTC m=+328.033706674" watchObservedRunningTime="2026-07-26 15:55:35.93089024 +0000 UTC m=+328.035116050"
Jul 26 15:55:35.944812 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:35.944735    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t5q6d" podStartSLOduration=2.377187297 podStartE2EDuration="5.94471678s" totalImagesPullingTime="3.567529483s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:55:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:55:35.943977528 +0000 UTC m=+328.048203336" watchObservedRunningTime="2026-07-26 15:55:35.94471678 +0000 UTC m=+328.048942589"
Jul 26 15:55:37.936283 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:37.936241    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 26 15:55:37.936565 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:37.936313    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 26 15:55:38.854689 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:38.854647    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:55:38.924348 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:38.924310    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5mp5d" event={"ID":"9d5f0889-c6a8-4641-b6f4-28e81a9a7955","Type":"ContainerStarted","Data":"97c7d462407686cd0b48df530d7849be328d5b947990f1f761a2396c9c514728"}
Jul 26 15:55:38.924600 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:38.924564    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5mp5d"
Jul 26 15:55:38.926504 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:38.926472    2563 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-5mp5d container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 26 15:55:38.926609 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:38.926531    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5mp5d" podUID="9d5f0889-c6a8-4641-b6f4-28e81a9a7955" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 26 15:55:38.955141 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:38.955079    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5mp5d" podStartSLOduration=2.22856416 podStartE2EDuration="5.955059422s" totalImagesPullingTime="3.726495262s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:55:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:55:38.953457055 +0000 UTC m=+331.057682888" watchObservedRunningTime="2026-07-26 15:55:38.955059422 +0000 UTC m=+331.059285229"
Jul 26 15:55:38.958690 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:38.958653    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:55:39.540235 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:39.540198    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:55:39.745228 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:39.745185    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:55:39.844025 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:39.843947    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:55:39.930040 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:39.930012    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5mp5d"
Jul 26 15:55:52.981785 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:52.981726    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:55:53.073950 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:55:53.073900    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:56:06.907961 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:56:06.907884    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t5q6d"
Jul 26 15:56:06.910965 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:56:06.910940    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-7p94c"
Jul 26 15:56:19.710053 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:56:19.710018    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:56:21.715364 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:56:21.715332    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:56:21.815737 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:56:21.815698    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:56:21.915690 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:56:21.915656    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:56:49.268257 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:56:49.268213    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:56:52.649804 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:56:52.649756    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:56:53.009562 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:56:53.009529    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:56:53.106413 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:56:53.106369    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:56:53.218989 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:56:53.218944    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:56:53.319176 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:56:53.319085    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:56:53.726677 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:56:53.726644    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:56:53.828225 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:56:53.828190    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:56:55.068301 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:56:55.068251    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:56:55.170434 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:56:55.170402    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:56:56.858378 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:56:56.858342    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:56:59.898608 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:56:59.898574    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:57:01.125113 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:01.125048    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:57:01.152991 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:01.152952    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2hqtt"]
Jul 26 15:57:01.155790 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:01.155752    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2hqtt"
Jul 26 15:57:01.160885 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:01.160854    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-x8dzq\""
Jul 26 15:57:01.161032 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:01.160913    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 26 15:57:01.161032 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:01.160973    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 26 15:57:01.167265 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:01.167238    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2hqtt"]
Jul 26 15:57:01.182920 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:01.182889    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9706105a-b681-4165-a6b3-7a2d2e3895df-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2hqtt\" (UID: \"9706105a-b681-4165-a6b3-7a2d2e3895df\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2hqtt"
Jul 26 15:57:01.183112 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:01.182935    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95tvd\" (UniqueName: \"kubernetes.io/projected/9706105a-b681-4165-a6b3-7a2d2e3895df-kube-api-access-95tvd\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2hqtt\" (UID: \"9706105a-b681-4165-a6b3-7a2d2e3895df\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2hqtt"
Jul 26 15:57:01.283824 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:01.283787    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9706105a-b681-4165-a6b3-7a2d2e3895df-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2hqtt\" (UID: \"9706105a-b681-4165-a6b3-7a2d2e3895df\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2hqtt"
Jul 26 15:57:01.283992 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:01.283840    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95tvd\" (UniqueName: \"kubernetes.io/projected/9706105a-b681-4165-a6b3-7a2d2e3895df-kube-api-access-95tvd\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2hqtt\" (UID: \"9706105a-b681-4165-a6b3-7a2d2e3895df\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2hqtt"
Jul 26 15:57:01.284189 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:01.284171    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9706105a-b681-4165-a6b3-7a2d2e3895df-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2hqtt\" (UID: \"9706105a-b681-4165-a6b3-7a2d2e3895df\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2hqtt"
Jul 26 15:57:01.296241 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:01.296203    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95tvd\" (UniqueName: \"kubernetes.io/projected/9706105a-b681-4165-a6b3-7a2d2e3895df-kube-api-access-95tvd\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2hqtt\" (UID: \"9706105a-b681-4165-a6b3-7a2d2e3895df\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2hqtt"
Jul 26 15:57:01.466249 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:01.466161    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2hqtt"
Jul 26 15:57:01.602249 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:01.602205    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2hqtt"]
Jul 26 15:57:01.606007 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:57:01.605973    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9706105a_b681_4165_a6b3_7a2d2e3895df.slice/crio-26bff950b21226f07b8f995342847829e28b304331ae57a2050f5813b7f2d9b7 WatchSource:0}: Error finding container 26bff950b21226f07b8f995342847829e28b304331ae57a2050f5813b7f2d9b7: Status 404 returned error can't find the container with id 26bff950b21226f07b8f995342847829e28b304331ae57a2050f5813b7f2d9b7
Jul 26 15:57:02.254304 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:02.254272    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:57:02.283667 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:02.283624    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2hqtt" event={"ID":"9706105a-b681-4165-a6b3-7a2d2e3895df","Type":"ContainerStarted","Data":"26bff950b21226f07b8f995342847829e28b304331ae57a2050f5813b7f2d9b7"}
Jul 26 15:57:03.182114 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:03.182079    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:57:03.583152 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:03.583115    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:57:03.685108 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:03.685075    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:57:03.783400 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:03.783365    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:57:05.677183 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:05.677151    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:57:05.781618 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:05.781587    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:57:08.313433 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:08.313397    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2hqtt" event={"ID":"9706105a-b681-4165-a6b3-7a2d2e3895df","Type":"ContainerStarted","Data":"2fec41d4d59a76acc2bdbd467a1d1538b68095aa967adcaf9a2aa0ec0ae9beef"}
Jul 26 15:57:08.313902 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:08.313543    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2hqtt"
Jul 26 15:57:08.352295 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:08.352242    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2hqtt" podStartSLOduration=1.658432739 podStartE2EDuration="7.352225772s" totalImagesPullingTime="5.693793033s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:57:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:57:08.349806041 +0000 UTC m=+420.454031846" watchObservedRunningTime="2026-07-26 15:57:08.352225772 +0000 UTC m=+420.456451578"
Jul 26 15:57:10.034393 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:10.034353    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:57:10.280494 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:10.280464    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:57:11.103521 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:11.103486    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:57:11.206679 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:11.206650    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:57:11.299076 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:11.299042    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:57:11.399716 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:11.399639    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:57:13.839029 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:13.838995    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:57:13.937597 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:13.937565    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:57:19.320493 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:19.320455    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2hqtt"
Jul 26 15:57:20.378508 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:20.378471    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-79snl"]
Jul 26 15:57:20.381845 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:20.381829    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-79snl"
Jul 26 15:57:20.394811 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:20.394787    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-79snl"]
Jul 26 15:57:20.452271 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:20.452238    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eef635e5-57b5-41a5-a07d-30e6bc66ad8e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-79snl\" (UID: \"eef635e5-57b5-41a5-a07d-30e6bc66ad8e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-79snl"
Jul 26 15:57:20.452514 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:20.452486    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lvbb\" (UniqueName: \"kubernetes.io/projected/eef635e5-57b5-41a5-a07d-30e6bc66ad8e-kube-api-access-5lvbb\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-79snl\" (UID: \"eef635e5-57b5-41a5-a07d-30e6bc66ad8e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-79snl"
Jul 26 15:57:20.553625 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:20.553582    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lvbb\" (UniqueName: \"kubernetes.io/projected/eef635e5-57b5-41a5-a07d-30e6bc66ad8e-kube-api-access-5lvbb\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-79snl\" (UID: \"eef635e5-57b5-41a5-a07d-30e6bc66ad8e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-79snl"
Jul 26 15:57:20.553824 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:20.553634    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eef635e5-57b5-41a5-a07d-30e6bc66ad8e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-79snl\" (UID: \"eef635e5-57b5-41a5-a07d-30e6bc66ad8e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-79snl"
Jul 26 15:57:20.553992 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:20.553976    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eef635e5-57b5-41a5-a07d-30e6bc66ad8e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-79snl\" (UID: \"eef635e5-57b5-41a5-a07d-30e6bc66ad8e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-79snl"
Jul 26 15:57:20.561688 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:20.561657    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lvbb\" (UniqueName: \"kubernetes.io/projected/eef635e5-57b5-41a5-a07d-30e6bc66ad8e-kube-api-access-5lvbb\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-79snl\" (UID: \"eef635e5-57b5-41a5-a07d-30e6bc66ad8e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-79snl"
Jul 26 15:57:20.692098 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:20.692008    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-79snl"
Jul 26 15:57:20.823985 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:20.823954    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-79snl"]
Jul 26 15:57:20.828211 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:57:20.828178    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeef635e5_57b5_41a5_a07d_30e6bc66ad8e.slice/crio-1a52ca931319a9ab50077cb33673629353d868289ffe08fa7041149de16c7583 WatchSource:0}: Error finding container 1a52ca931319a9ab50077cb33673629353d868289ffe08fa7041149de16c7583: Status 404 returned error can't find the container with id 1a52ca931319a9ab50077cb33673629353d868289ffe08fa7041149de16c7583
Jul 26 15:57:20.886780 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:20.876842    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:57:20.984054 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:20.984023    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:57:21.095230 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:21.095189    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-79snl"]
Jul 26 15:57:21.099849 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:21.099823    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-79snl"]
Jul 26 15:57:21.112499 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:21.112472    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2hqtt"]
Jul 26 15:57:21.112675 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:21.112655    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2hqtt" podUID="9706105a-b681-4165-a6b3-7a2d2e3895df" containerName="manager" containerID="cri-o://2fec41d4d59a76acc2bdbd467a1d1538b68095aa967adcaf9a2aa0ec0ae9beef" gracePeriod=2
Jul 26 15:57:21.118443 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:21.117943    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xwscr"]
Jul 26 15:57:21.118443 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:21.118389    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eef635e5-57b5-41a5-a07d-30e6bc66ad8e" containerName="manager"
Jul 26 15:57:21.118443 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:21.118408    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="eef635e5-57b5-41a5-a07d-30e6bc66ad8e" containerName="manager"
Jul 26 15:57:21.118680 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:21.118507    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="eef635e5-57b5-41a5-a07d-30e6bc66ad8e" containerName="manager"
Jul 26 15:57:21.122494 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:21.122468    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2hqtt"]
Jul 26 15:57:21.122621 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:21.122567    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xwscr"
Jul 26 15:57:21.136104 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:21.136077    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xwscr"]
Jul 26 15:57:21.139872 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:21.139846    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bk5mq"]
Jul 26 15:57:21.140161 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:21.140149    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9706105a-b681-4165-a6b3-7a2d2e3895df" containerName="manager"
Jul 26 15:57:21.140207 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:21.140164    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9706105a-b681-4165-a6b3-7a2d2e3895df" containerName="manager"
Jul 26 15:57:21.140245 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:21.140233    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9706105a-b681-4165-a6b3-7a2d2e3895df" containerName="manager"
Jul 26 15:57:21.143146 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:21.143126    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bk5mq"
Jul 26 15:57:21.158914 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:21.158890    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjxm5\" (UniqueName: \"kubernetes.io/projected/2c5ba680-f536-4884-b9be-6d6cdeb6da3f-kube-api-access-kjxm5\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-xwscr\" (UID: \"2c5ba680-f536-4884-b9be-6d6cdeb6da3f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xwscr"
Jul 26 15:57:21.159012 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:21.158934    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2c5ba680-f536-4884-b9be-6d6cdeb6da3f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-xwscr\" (UID: \"2c5ba680-f536-4884-b9be-6d6cdeb6da3f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xwscr"
Jul 26 15:57:21.163459 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:21.163431    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bk5mq"]
Jul 26 15:57:21.164348 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:57:21.164317    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-2hqtt\" is forbidden: User \"system:node:ip-10-0-133-204.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-204.ec2.internal' and this object" podUID="9706105a-b681-4165-a6b3-7a2d2e3895df" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2hqtt"
Jul 26 15:57:21.166063 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:57:21.166036    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-2hqtt\" is forbidden: User \"system:node:ip-10-0-133-204.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-204.ec2.internal' and this object" podUID="9706105a-b681-4165-a6b3-7a2d2e3895df" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2hqtt"
Jul 26 15:57:21.260041 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:21.259898    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2c5ba680-f536-4884-b9be-6d6cdeb6da3f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-xwscr\" (UID: \"2c5ba680-f536-4884-b9be-6d6cdeb6da3f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xwscr"
Jul 26 15:57:21.260184 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:21.260061    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f0582af6-3336-48dd-a4f1-eeff2b731918-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-bk5mq\" (UID: \"f0582af6-3336-48dd-a4f1-eeff2b731918\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bk5mq"
Jul 26 15:57:21.260184 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:21.260106    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmhnd\" (UniqueName: \"kubernetes.io/projected/f0582af6-3336-48dd-a4f1-eeff2b731918-kube-api-access-mmhnd\") pod \"kuadrant-operator-controller-manager-8684c9ff49-bk5mq\" (UID: \"f0582af6-3336-48dd-a4f1-eeff2b731918\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bk5mq"
Jul 26 15:57:21.260184 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:21.260178    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjxm5\" (UniqueName: \"kubernetes.io/projected/2c5ba680-f536-4884-b9be-6d6cdeb6da3f-kube-api-access-kjxm5\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-xwscr\" (UID: \"2c5ba680-f536-4884-b9be-6d6cdeb6da3f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xwscr"
Jul 26 15:57:21.260367 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:21.260351    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2c5ba680-f536-4884-b9be-6d6cdeb6da3f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-xwscr\" (UID: \"2c5ba680-f536-4884-b9be-6d6cdeb6da3f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xwscr"
Jul 26 15:57:21.268031 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:21.268009    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjxm5\" (UniqueName: \"kubernetes.io/projected/2c5ba680-f536-4884-b9be-6d6cdeb6da3f-kube-api-access-kjxm5\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-xwscr\" (UID: \"2c5ba680-f536-4884-b9be-6d6cdeb6da3f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xwscr"
Jul 26 15:57:21.347594 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:21.347575    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2hqtt"
Jul 26 15:57:21.349751 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:57:21.349725    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-2hqtt\" is forbidden: User \"system:node:ip-10-0-133-204.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-204.ec2.internal' and this object" podUID="9706105a-b681-4165-a6b3-7a2d2e3895df" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2hqtt"
Jul 26 15:57:21.361150 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:21.361126    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f0582af6-3336-48dd-a4f1-eeff2b731918-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-bk5mq\" (UID: \"f0582af6-3336-48dd-a4f1-eeff2b731918\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bk5mq"
Jul 26 15:57:21.361221 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:21.361154    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmhnd\" (UniqueName: \"kubernetes.io/projected/f0582af6-3336-48dd-a4f1-eeff2b731918-kube-api-access-mmhnd\") pod \"kuadrant-operator-controller-manager-8684c9ff49-bk5mq\" (UID: \"f0582af6-3336-48dd-a4f1-eeff2b731918\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bk5mq"
Jul 26 15:57:21.361531 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:21.361512    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f0582af6-3336-48dd-a4f1-eeff2b731918-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-bk5mq\" (UID: \"f0582af6-3336-48dd-a4f1-eeff2b731918\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bk5mq"
Jul 26 15:57:21.369164 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:21.369144    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmhnd\" (UniqueName: \"kubernetes.io/projected/f0582af6-3336-48dd-a4f1-eeff2b731918-kube-api-access-mmhnd\") pod \"kuadrant-operator-controller-manager-8684c9ff49-bk5mq\" (UID: \"f0582af6-3336-48dd-a4f1-eeff2b731918\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bk5mq"
Jul 26 15:57:21.374877 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:21.374852    2563 generic.go:356] "Generic (PLEG): container finished" podID="9706105a-b681-4165-a6b3-7a2d2e3895df" containerID="2fec41d4d59a76acc2bdbd467a1d1538b68095aa967adcaf9a2aa0ec0ae9beef" exitCode=0
Jul 26 15:57:21.374993 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:21.374905    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2hqtt"
Jul 26 15:57:21.374993 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:21.374952    2563 scope.go:117] "RemoveContainer" containerID="2fec41d4d59a76acc2bdbd467a1d1538b68095aa967adcaf9a2aa0ec0ae9beef"
Jul 26 15:57:21.376878 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:57:21.376860    2563 kuberuntime_manager.go:649] "Missing actuated resource record" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-79snl" container="manager"
Jul 26 15:57:21.377077 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:57:21.377051    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-2hqtt\" is forbidden: User \"system:node:ip-10-0-133-204.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-204.ec2.internal' and this object" podUID="9706105a-b681-4165-a6b3-7a2d2e3895df" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2hqtt"
Jul 26 15:57:21.378922 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:57:21.378893    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-2hqtt\" is forbidden: User \"system:node:ip-10-0-133-204.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-204.ec2.internal' and this object" podUID="9706105a-b681-4165-a6b3-7a2d2e3895df" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2hqtt"
Jul 26 15:57:21.380678 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:57:21.380655    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-79snl\" is forbidden: User \"system:node:ip-10-0-133-204.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-204.ec2.internal' and this object" podUID="eef635e5-57b5-41a5-a07d-30e6bc66ad8e" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-79snl"
Jul 26 15:57:21.384009 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:21.383990    2563 scope.go:117] "RemoveContainer" containerID="2fec41d4d59a76acc2bdbd467a1d1538b68095aa967adcaf9a2aa0ec0ae9beef"
Jul 26 15:57:21.384266 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:57:21.384244    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fec41d4d59a76acc2bdbd467a1d1538b68095aa967adcaf9a2aa0ec0ae9beef\": container with ID starting with 2fec41d4d59a76acc2bdbd467a1d1538b68095aa967adcaf9a2aa0ec0ae9beef not found: ID does not exist" containerID="2fec41d4d59a76acc2bdbd467a1d1538b68095aa967adcaf9a2aa0ec0ae9beef"
Jul 26 15:57:21.384343 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:21.384278    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fec41d4d59a76acc2bdbd467a1d1538b68095aa967adcaf9a2aa0ec0ae9beef"} err="failed to get container status \"2fec41d4d59a76acc2bdbd467a1d1538b68095aa967adcaf9a2aa0ec0ae9beef\": rpc error: code = NotFound desc = could not find container \"2fec41d4d59a76acc2bdbd467a1d1538b68095aa967adcaf9a2aa0ec0ae9beef\": container with ID starting with 2fec41d4d59a76acc2bdbd467a1d1538b68095aa967adcaf9a2aa0ec0ae9beef not found: ID does not exist"
Jul 26 15:57:21.461668 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:21.461635    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-95tvd\" (UniqueName: \"kubernetes.io/projected/9706105a-b681-4165-a6b3-7a2d2e3895df-kube-api-access-95tvd\") pod \"9706105a-b681-4165-a6b3-7a2d2e3895df\" (UID: \"9706105a-b681-4165-a6b3-7a2d2e3895df\") "
Jul 26 15:57:21.461881 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:21.461689    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9706105a-b681-4165-a6b3-7a2d2e3895df-extensions-socket-volume\") pod \"9706105a-b681-4165-a6b3-7a2d2e3895df\" (UID: \"9706105a-b681-4165-a6b3-7a2d2e3895df\") "
Jul 26 15:57:21.462129 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:21.462095    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9706105a-b681-4165-a6b3-7a2d2e3895df-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "9706105a-b681-4165-a6b3-7a2d2e3895df" (UID: "9706105a-b681-4165-a6b3-7a2d2e3895df"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 26 15:57:21.463805 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:21.463781    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9706105a-b681-4165-a6b3-7a2d2e3895df-kube-api-access-95tvd" (OuterVolumeSpecName: "kube-api-access-95tvd") pod "9706105a-b681-4165-a6b3-7a2d2e3895df" (UID: "9706105a-b681-4165-a6b3-7a2d2e3895df"). InnerVolumeSpecName "kube-api-access-95tvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 26 15:57:21.479932 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:21.479909    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xwscr"
Jul 26 15:57:21.487890 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:21.487866    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bk5mq"
Jul 26 15:57:21.562985 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:21.562913    2563 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9706105a-b681-4165-a6b3-7a2d2e3895df-extensions-socket-volume\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 15:57:21.562985 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:21.562949    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95tvd\" (UniqueName: \"kubernetes.io/projected/9706105a-b681-4165-a6b3-7a2d2e3895df-kube-api-access-95tvd\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 15:57:21.629687 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:21.629652    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xwscr"]
Jul 26 15:57:21.634442 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:57:21.634414    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c5ba680_f536_4884_b9be_6d6cdeb6da3f.slice/crio-ec2640b28ecfe653ac159149237275255f150b669941753f8543767af959cd63 WatchSource:0}: Error finding container ec2640b28ecfe653ac159149237275255f150b669941753f8543767af959cd63: Status 404 returned error can't find the container with id ec2640b28ecfe653ac159149237275255f150b669941753f8543767af959cd63
Jul 26 15:57:21.646574 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:21.646553    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bk5mq"]
Jul 26 15:57:21.652753 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:57:21.652692    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0582af6_3336_48dd_a4f1_eeff2b731918.slice/crio-a011719b21926983877d52302b411fe66c099cdba089e8886a5ae803009f0a8c WatchSource:0}: Error finding container a011719b21926983877d52302b411fe66c099cdba089e8886a5ae803009f0a8c: Status 404 returned error can't find the container with id a011719b21926983877d52302b411fe66c099cdba089e8886a5ae803009f0a8c
Jul 26 15:57:21.711608 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:57:21.711576    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-79snl\" is forbidden: User \"system:node:ip-10-0-133-204.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-204.ec2.internal' and this object" podUID="eef635e5-57b5-41a5-a07d-30e6bc66ad8e" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-79snl"
Jul 26 15:57:21.713707 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:57:21.713682    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-2hqtt\" is forbidden: User \"system:node:ip-10-0-133-204.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-204.ec2.internal' and this object" podUID="9706105a-b681-4165-a6b3-7a2d2e3895df" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2hqtt"
Jul 26 15:57:22.372735 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:22.372694    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9706105a-b681-4165-a6b3-7a2d2e3895df" path="/var/lib/kubelet/pods/9706105a-b681-4165-a6b3-7a2d2e3895df/volumes"
Jul 26 15:57:22.382489 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:22.382456    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-79snl" podUID="eef635e5-57b5-41a5-a07d-30e6bc66ad8e" containerName="manager" containerID="cri-o://3363abba9702e673262db55ac48b63f4faeac27be86b8af6c7d8e27ceb4b044a" gracePeriod=2
Jul 26 15:57:22.384060 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:22.384028    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xwscr" event={"ID":"2c5ba680-f536-4884-b9be-6d6cdeb6da3f","Type":"ContainerStarted","Data":"2f8f59a642b35739058bb0f3f8d3691ecc082015c46dad8e2fcf4d1746725d1a"}
Jul 26 15:57:22.384182 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:22.384090    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xwscr"
Jul 26 15:57:22.384182 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:22.384116    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xwscr" event={"ID":"2c5ba680-f536-4884-b9be-6d6cdeb6da3f","Type":"ContainerStarted","Data":"ec2640b28ecfe653ac159149237275255f150b669941753f8543767af959cd63"}
Jul 26 15:57:22.385893 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:22.385861    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bk5mq" event={"ID":"f0582af6-3336-48dd-a4f1-eeff2b731918","Type":"ContainerStarted","Data":"a4993cf6636787aa2cc37c9b155d9d88e473afc3832dbbdf2b8499531d628334"}
Jul 26 15:57:22.386039 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:22.385918    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bk5mq" event={"ID":"f0582af6-3336-48dd-a4f1-eeff2b731918","Type":"ContainerStarted","Data":"a011719b21926983877d52302b411fe66c099cdba089e8886a5ae803009f0a8c"}
Jul 26 15:57:22.386421 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:22.386394    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bk5mq"
Jul 26 15:57:22.405463 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:22.405413    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xwscr" podStartSLOduration=1.405398151 podStartE2EDuration="1.405398151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:57:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:57:22.403956975 +0000 UTC m=+434.508182836" watchObservedRunningTime="2026-07-26 15:57:22.405398151 +0000 UTC m=+434.509624002"
Jul 26 15:57:22.420994 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:22.420950    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bk5mq" podStartSLOduration=1.420934653 podStartE2EDuration="1.420934653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:57:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:57:22.420089617 +0000 UTC m=+434.524315463" watchObservedRunningTime="2026-07-26 15:57:22.420934653 +0000 UTC m=+434.525160504"
Jul 26 15:57:22.623310 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:22.623254    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-79snl"
Jul 26 15:57:22.628334 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:57:22.628305    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-79snl\" is forbidden: User \"system:node:ip-10-0-133-204.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-204.ec2.internal' and this object" podUID="eef635e5-57b5-41a5-a07d-30e6bc66ad8e" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-79snl"
Jul 26 15:57:22.672821 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:22.672787    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5lvbb\" (UniqueName: \"kubernetes.io/projected/eef635e5-57b5-41a5-a07d-30e6bc66ad8e-kube-api-access-5lvbb\") pod \"eef635e5-57b5-41a5-a07d-30e6bc66ad8e\" (UID: \"eef635e5-57b5-41a5-a07d-30e6bc66ad8e\") "
Jul 26 15:57:22.672984 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:22.672854    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eef635e5-57b5-41a5-a07d-30e6bc66ad8e-extensions-socket-volume\") pod \"eef635e5-57b5-41a5-a07d-30e6bc66ad8e\" (UID: \"eef635e5-57b5-41a5-a07d-30e6bc66ad8e\") "
Jul 26 15:57:22.673225 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:22.673200    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eef635e5-57b5-41a5-a07d-30e6bc66ad8e-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "eef635e5-57b5-41a5-a07d-30e6bc66ad8e" (UID: "eef635e5-57b5-41a5-a07d-30e6bc66ad8e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 26 15:57:22.674949 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:22.674928    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eef635e5-57b5-41a5-a07d-30e6bc66ad8e-kube-api-access-5lvbb" (OuterVolumeSpecName: "kube-api-access-5lvbb") pod "eef635e5-57b5-41a5-a07d-30e6bc66ad8e" (UID: "eef635e5-57b5-41a5-a07d-30e6bc66ad8e"). InnerVolumeSpecName "kube-api-access-5lvbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 26 15:57:22.774033 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:22.773992    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lvbb\" (UniqueName: \"kubernetes.io/projected/eef635e5-57b5-41a5-a07d-30e6bc66ad8e-kube-api-access-5lvbb\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 15:57:22.774033 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:22.774028    2563 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eef635e5-57b5-41a5-a07d-30e6bc66ad8e-extensions-socket-volume\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 15:57:23.391490 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:23.391453    2563 generic.go:356] "Generic (PLEG): container finished" podID="eef635e5-57b5-41a5-a07d-30e6bc66ad8e" containerID="3363abba9702e673262db55ac48b63f4faeac27be86b8af6c7d8e27ceb4b044a" exitCode=0
Jul 26 15:57:23.391490 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:23.391495    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-79snl"
Jul 26 15:57:23.392013 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:23.391554    2563 scope.go:117] "RemoveContainer" containerID="3363abba9702e673262db55ac48b63f4faeac27be86b8af6c7d8e27ceb4b044a"
Jul 26 15:57:23.394128 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:57:23.394101    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-79snl\" is forbidden: User \"system:node:ip-10-0-133-204.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-204.ec2.internal' and this object" podUID="eef635e5-57b5-41a5-a07d-30e6bc66ad8e" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-79snl"
Jul 26 15:57:23.400799 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:23.400753    2563 scope.go:117] "RemoveContainer" containerID="3363abba9702e673262db55ac48b63f4faeac27be86b8af6c7d8e27ceb4b044a"
Jul 26 15:57:23.401091 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:57:23.401072    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3363abba9702e673262db55ac48b63f4faeac27be86b8af6c7d8e27ceb4b044a\": container with ID starting with 3363abba9702e673262db55ac48b63f4faeac27be86b8af6c7d8e27ceb4b044a not found: ID does not exist" containerID="3363abba9702e673262db55ac48b63f4faeac27be86b8af6c7d8e27ceb4b044a"
Jul 26 15:57:23.401150 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:23.401098    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3363abba9702e673262db55ac48b63f4faeac27be86b8af6c7d8e27ceb4b044a"} err="failed to get container status \"3363abba9702e673262db55ac48b63f4faeac27be86b8af6c7d8e27ceb4b044a\": rpc error: code = NotFound desc = could not find container \"3363abba9702e673262db55ac48b63f4faeac27be86b8af6c7d8e27ceb4b044a\": container with ID starting with 3363abba9702e673262db55ac48b63f4faeac27be86b8af6c7d8e27ceb4b044a not found: ID does not exist"
Jul 26 15:57:23.401921 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:57:23.401901    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-79snl\" is forbidden: User \"system:node:ip-10-0-133-204.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-204.ec2.internal' and this object" podUID="eef635e5-57b5-41a5-a07d-30e6bc66ad8e" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-79snl"
Jul 26 15:57:23.508529 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:23.508492    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:57:23.608459 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:23.608425    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:57:24.372520 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:24.372485    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eef635e5-57b5-41a5-a07d-30e6bc66ad8e" path="/var/lib/kubelet/pods/eef635e5-57b5-41a5-a07d-30e6bc66ad8e/volumes"
Jul 26 15:57:27.464131 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:27.464100    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:57:27.531746 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:27.531715    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:57:28.867621 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:28.867584    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:57:28.968982 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:28.968952    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:57:29.467659 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:29.467629    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:57:29.520115 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:29.520081    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:57:33.073803 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:33.073699    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:57:33.128384 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:33.128344    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:57:33.393629 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:33.393542    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xwscr"
Jul 26 15:57:33.453509 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:33.453477    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bk5mq"]
Jul 26 15:57:33.453696 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:33.453668    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bk5mq" podUID="f0582af6-3336-48dd-a4f1-eeff2b731918" containerName="manager" containerID="cri-o://a4993cf6636787aa2cc37c9b155d9d88e473afc3832dbbdf2b8499531d628334" gracePeriod=10
Jul 26 15:57:33.455701 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:33.455681    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bk5mq"
Jul 26 15:57:33.705397 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:33.705374    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bk5mq"
Jul 26 15:57:33.769151 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:33.768980    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mmhnd\" (UniqueName: \"kubernetes.io/projected/f0582af6-3336-48dd-a4f1-eeff2b731918-kube-api-access-mmhnd\") pod \"f0582af6-3336-48dd-a4f1-eeff2b731918\" (UID: \"f0582af6-3336-48dd-a4f1-eeff2b731918\") "
Jul 26 15:57:33.769151 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:33.769067    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f0582af6-3336-48dd-a4f1-eeff2b731918-extensions-socket-volume\") pod \"f0582af6-3336-48dd-a4f1-eeff2b731918\" (UID: \"f0582af6-3336-48dd-a4f1-eeff2b731918\") "
Jul 26 15:57:33.769753 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:33.769727    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0582af6-3336-48dd-a4f1-eeff2b731918-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "f0582af6-3336-48dd-a4f1-eeff2b731918" (UID: "f0582af6-3336-48dd-a4f1-eeff2b731918"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 26 15:57:33.771449 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:33.771422    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2wt74"]
Jul 26 15:57:33.771782 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:33.771741    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0582af6-3336-48dd-a4f1-eeff2b731918-kube-api-access-mmhnd" (OuterVolumeSpecName: "kube-api-access-mmhnd") pod "f0582af6-3336-48dd-a4f1-eeff2b731918" (UID: "f0582af6-3336-48dd-a4f1-eeff2b731918"). InnerVolumeSpecName "kube-api-access-mmhnd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 26 15:57:33.771854 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:33.771807    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0582af6-3336-48dd-a4f1-eeff2b731918" containerName="manager"
Jul 26 15:57:33.771854 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:33.771818    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0582af6-3336-48dd-a4f1-eeff2b731918" containerName="manager"
Jul 26 15:57:33.771923 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:33.771894    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0582af6-3336-48dd-a4f1-eeff2b731918" containerName="manager"
Jul 26 15:57:33.774974 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:33.774960    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2wt74"
Jul 26 15:57:33.792328 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:33.792305    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2wt74"]
Jul 26 15:57:33.870551 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:33.870516    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6ac3f9b7-ebaf-41ad-ba2d-70c2ef6e81d3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-2wt74\" (UID: \"6ac3f9b7-ebaf-41ad-ba2d-70c2ef6e81d3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2wt74"
Jul 26 15:57:33.870726 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:33.870578    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-664qz\" (UniqueName: \"kubernetes.io/projected/6ac3f9b7-ebaf-41ad-ba2d-70c2ef6e81d3-kube-api-access-664qz\") pod \"kuadrant-operator-controller-manager-86878c769f-2wt74\" (UID: \"6ac3f9b7-ebaf-41ad-ba2d-70c2ef6e81d3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2wt74"
Jul 26 15:57:33.870726 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:33.870649    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmhnd\" (UniqueName: \"kubernetes.io/projected/f0582af6-3336-48dd-a4f1-eeff2b731918-kube-api-access-mmhnd\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 15:57:33.870726 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:33.870661    2563 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f0582af6-3336-48dd-a4f1-eeff2b731918-extensions-socket-volume\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 15:57:33.971526 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:33.971428    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-664qz\" (UniqueName: \"kubernetes.io/projected/6ac3f9b7-ebaf-41ad-ba2d-70c2ef6e81d3-kube-api-access-664qz\") pod \"kuadrant-operator-controller-manager-86878c769f-2wt74\" (UID: \"6ac3f9b7-ebaf-41ad-ba2d-70c2ef6e81d3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2wt74"
Jul 26 15:57:33.971526 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:33.971518    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6ac3f9b7-ebaf-41ad-ba2d-70c2ef6e81d3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-2wt74\" (UID: \"6ac3f9b7-ebaf-41ad-ba2d-70c2ef6e81d3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2wt74"
Jul 26 15:57:33.971913 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:33.971897    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6ac3f9b7-ebaf-41ad-ba2d-70c2ef6e81d3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-2wt74\" (UID: \"6ac3f9b7-ebaf-41ad-ba2d-70c2ef6e81d3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2wt74"
Jul 26 15:57:33.979876 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:33.979853    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-664qz\" (UniqueName: \"kubernetes.io/projected/6ac3f9b7-ebaf-41ad-ba2d-70c2ef6e81d3-kube-api-access-664qz\") pod \"kuadrant-operator-controller-manager-86878c769f-2wt74\" (UID: \"6ac3f9b7-ebaf-41ad-ba2d-70c2ef6e81d3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2wt74"
Jul 26 15:57:34.087631 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:34.087593    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2wt74"
Jul 26 15:57:34.227811 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:34.227785    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2wt74"]
Jul 26 15:57:34.230321 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:57:34.230295    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ac3f9b7_ebaf_41ad_ba2d_70c2ef6e81d3.slice/crio-d603213b2e4ea782cb8238fe71ff8e1db35be47ba2f9152437a767cf796add85 WatchSource:0}: Error finding container d603213b2e4ea782cb8238fe71ff8e1db35be47ba2f9152437a767cf796add85: Status 404 returned error can't find the container with id d603213b2e4ea782cb8238fe71ff8e1db35be47ba2f9152437a767cf796add85
Jul 26 15:57:34.440375 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:34.440339    2563 generic.go:356] "Generic (PLEG): container finished" podID="f0582af6-3336-48dd-a4f1-eeff2b731918" containerID="a4993cf6636787aa2cc37c9b155d9d88e473afc3832dbbdf2b8499531d628334" exitCode=0
Jul 26 15:57:34.440574 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:34.440388    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bk5mq"
Jul 26 15:57:34.440574 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:34.440424    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bk5mq" event={"ID":"f0582af6-3336-48dd-a4f1-eeff2b731918","Type":"ContainerDied","Data":"a4993cf6636787aa2cc37c9b155d9d88e473afc3832dbbdf2b8499531d628334"}
Jul 26 15:57:34.440574 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:34.440473    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bk5mq" event={"ID":"f0582af6-3336-48dd-a4f1-eeff2b731918","Type":"ContainerDied","Data":"a011719b21926983877d52302b411fe66c099cdba089e8886a5ae803009f0a8c"}
Jul 26 15:57:34.440574 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:34.440506    2563 scope.go:117] "RemoveContainer" containerID="a4993cf6636787aa2cc37c9b155d9d88e473afc3832dbbdf2b8499531d628334"
Jul 26 15:57:34.442191 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:34.442163    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2wt74" event={"ID":"6ac3f9b7-ebaf-41ad-ba2d-70c2ef6e81d3","Type":"ContainerStarted","Data":"64365ac7382326182fec1d258749b919ae7357cf895a0ee6e97d863ecd01958b"}
Jul 26 15:57:34.442302 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:34.442210    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2wt74" event={"ID":"6ac3f9b7-ebaf-41ad-ba2d-70c2ef6e81d3","Type":"ContainerStarted","Data":"d603213b2e4ea782cb8238fe71ff8e1db35be47ba2f9152437a767cf796add85"}
Jul 26 15:57:34.442423 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:34.442309    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2wt74"
Jul 26 15:57:34.449033 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:34.449016    2563 scope.go:117] "RemoveContainer" containerID="a4993cf6636787aa2cc37c9b155d9d88e473afc3832dbbdf2b8499531d628334"
Jul 26 15:57:34.449291 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:57:34.449273    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4993cf6636787aa2cc37c9b155d9d88e473afc3832dbbdf2b8499531d628334\": container with ID starting with a4993cf6636787aa2cc37c9b155d9d88e473afc3832dbbdf2b8499531d628334 not found: ID does not exist" containerID="a4993cf6636787aa2cc37c9b155d9d88e473afc3832dbbdf2b8499531d628334"
Jul 26 15:57:34.449339 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:34.449299    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4993cf6636787aa2cc37c9b155d9d88e473afc3832dbbdf2b8499531d628334"} err="failed to get container status \"a4993cf6636787aa2cc37c9b155d9d88e473afc3832dbbdf2b8499531d628334\": rpc error: code = NotFound desc = could not find container \"a4993cf6636787aa2cc37c9b155d9d88e473afc3832dbbdf2b8499531d628334\": container with ID starting with a4993cf6636787aa2cc37c9b155d9d88e473afc3832dbbdf2b8499531d628334 not found: ID does not exist"
Jul 26 15:57:34.465555 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:34.465517    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2wt74" podStartSLOduration=1.465505432 podStartE2EDuration="1.465505432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:57:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:57:34.46471287 +0000 UTC m=+446.568938681" watchObservedRunningTime="2026-07-26 15:57:34.465505432 +0000 UTC m=+446.569731321"
Jul 26 15:57:34.481072 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:34.481009    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bk5mq"]
Jul 26 15:57:34.490045 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:34.490024    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bk5mq"]
Jul 26 15:57:36.372989 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:36.372956    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0582af6-3336-48dd-a4f1-eeff2b731918" path="/var/lib/kubelet/pods/f0582af6-3336-48dd-a4f1-eeff2b731918/volumes"
Jul 26 15:57:39.677898 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:39.677863    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:57:39.778135 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:39.778101    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:57:41.783741 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:41.783702    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:57:41.842068 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:41.842040    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:57:45.450662 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:45.450631    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2wt74"
Jul 26 15:57:45.499499 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:45.499472    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xwscr"]
Jul 26 15:57:45.499699 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:45.499677    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xwscr" podUID="2c5ba680-f536-4884-b9be-6d6cdeb6da3f" containerName="manager" containerID="cri-o://2f8f59a642b35739058bb0f3f8d3691ecc082015c46dad8e2fcf4d1746725d1a" gracePeriod=10
Jul 26 15:57:45.746737 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:45.746713    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xwscr"
Jul 26 15:57:45.874865 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:45.874836    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2c5ba680-f536-4884-b9be-6d6cdeb6da3f-extensions-socket-volume\") pod \"2c5ba680-f536-4884-b9be-6d6cdeb6da3f\" (UID: \"2c5ba680-f536-4884-b9be-6d6cdeb6da3f\") "
Jul 26 15:57:45.875037 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:45.874873    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kjxm5\" (UniqueName: \"kubernetes.io/projected/2c5ba680-f536-4884-b9be-6d6cdeb6da3f-kube-api-access-kjxm5\") pod \"2c5ba680-f536-4884-b9be-6d6cdeb6da3f\" (UID: \"2c5ba680-f536-4884-b9be-6d6cdeb6da3f\") "
Jul 26 15:57:45.875188 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:45.875166    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c5ba680-f536-4884-b9be-6d6cdeb6da3f-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "2c5ba680-f536-4884-b9be-6d6cdeb6da3f" (UID: "2c5ba680-f536-4884-b9be-6d6cdeb6da3f"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 26 15:57:45.876962 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:45.876938    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c5ba680-f536-4884-b9be-6d6cdeb6da3f-kube-api-access-kjxm5" (OuterVolumeSpecName: "kube-api-access-kjxm5") pod "2c5ba680-f536-4884-b9be-6d6cdeb6da3f" (UID: "2c5ba680-f536-4884-b9be-6d6cdeb6da3f"). InnerVolumeSpecName "kube-api-access-kjxm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 26 15:57:45.976319 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:45.976284    2563 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2c5ba680-f536-4884-b9be-6d6cdeb6da3f-extensions-socket-volume\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 15:57:45.976319 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:45.976311    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjxm5\" (UniqueName: \"kubernetes.io/projected/2c5ba680-f536-4884-b9be-6d6cdeb6da3f-kube-api-access-kjxm5\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 15:57:46.498305 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:46.498274    2563 generic.go:356] "Generic (PLEG): container finished" podID="2c5ba680-f536-4884-b9be-6d6cdeb6da3f" containerID="2f8f59a642b35739058bb0f3f8d3691ecc082015c46dad8e2fcf4d1746725d1a" exitCode=0
Jul 26 15:57:46.498691 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:46.498328    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xwscr"
Jul 26 15:57:46.498691 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:46.498327    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xwscr" event={"ID":"2c5ba680-f536-4884-b9be-6d6cdeb6da3f","Type":"ContainerDied","Data":"2f8f59a642b35739058bb0f3f8d3691ecc082015c46dad8e2fcf4d1746725d1a"}
Jul 26 15:57:46.498691 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:46.498392    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xwscr" event={"ID":"2c5ba680-f536-4884-b9be-6d6cdeb6da3f","Type":"ContainerDied","Data":"ec2640b28ecfe653ac159149237275255f150b669941753f8543767af959cd63"}
Jul 26 15:57:46.498691 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:46.498422    2563 scope.go:117] "RemoveContainer" containerID="2f8f59a642b35739058bb0f3f8d3691ecc082015c46dad8e2fcf4d1746725d1a"
Jul 26 15:57:46.507178 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:46.506989    2563 scope.go:117] "RemoveContainer" containerID="2f8f59a642b35739058bb0f3f8d3691ecc082015c46dad8e2fcf4d1746725d1a"
Jul 26 15:57:46.507261 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:57:46.507243    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f8f59a642b35739058bb0f3f8d3691ecc082015c46dad8e2fcf4d1746725d1a\": container with ID starting with 2f8f59a642b35739058bb0f3f8d3691ecc082015c46dad8e2fcf4d1746725d1a not found: ID does not exist" containerID="2f8f59a642b35739058bb0f3f8d3691ecc082015c46dad8e2fcf4d1746725d1a"
Jul 26 15:57:46.507302 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:46.507269    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f8f59a642b35739058bb0f3f8d3691ecc082015c46dad8e2fcf4d1746725d1a"} err="failed to get container status \"2f8f59a642b35739058bb0f3f8d3691ecc082015c46dad8e2fcf4d1746725d1a\": rpc error: code = NotFound desc = could not find container \"2f8f59a642b35739058bb0f3f8d3691ecc082015c46dad8e2fcf4d1746725d1a\": container with ID starting with 2f8f59a642b35739058bb0f3f8d3691ecc082015c46dad8e2fcf4d1746725d1a not found: ID does not exist"
Jul 26 15:57:46.536083 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:46.536063    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xwscr"]
Jul 26 15:57:46.551902 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:46.551878    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xwscr"]
Jul 26 15:57:48.373365 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:48.373331    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c5ba680-f536-4884-b9be-6d6cdeb6da3f" path="/var/lib/kubelet/pods/2c5ba680-f536-4884-b9be-6d6cdeb6da3f/volumes"
Jul 26 15:57:49.821429 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:49.821359    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:57:50.139067 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:57:50.138996    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:58:03.452692 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:03.452662    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-9nzrt"]
Jul 26 15:58:03.453152 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:03.453010    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c5ba680-f536-4884-b9be-6d6cdeb6da3f" containerName="manager"
Jul 26 15:58:03.453152 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:03.453022    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c5ba680-f536-4884-b9be-6d6cdeb6da3f" containerName="manager"
Jul 26 15:58:03.453152 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:03.453067    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c5ba680-f536-4884-b9be-6d6cdeb6da3f" containerName="manager"
Jul 26 15:58:03.458025 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:03.457998    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-9nzrt"
Jul 26 15:58:03.458648 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:03.457991    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:58:03.461092 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:03.461067    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-bfkp9\""
Jul 26 15:58:03.461203 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:03.461067    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 26 15:58:03.465253 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:03.465231    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-9nzrt"]
Jul 26 15:58:03.529706 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:03.529676    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdlfp\" (UniqueName: \"kubernetes.io/projected/c638e86b-4d59-46fd-ad8c-49d5348889bd-kube-api-access-tdlfp\") pod \"limitador-limitador-64c94fd64f-9nzrt\" (UID: \"c638e86b-4d59-46fd-ad8c-49d5348889bd\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-9nzrt"
Jul 26 15:58:03.529893 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:03.529756    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c638e86b-4d59-46fd-ad8c-49d5348889bd-config-file\") pod \"limitador-limitador-64c94fd64f-9nzrt\" (UID: \"c638e86b-4d59-46fd-ad8c-49d5348889bd\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-9nzrt"
Jul 26 15:58:03.545745 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:03.545717    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-9nzrt"]
Jul 26 15:58:03.556746 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:03.556715    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:58:03.608155 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:03.608126    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:58:03.631209 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:03.631180    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c638e86b-4d59-46fd-ad8c-49d5348889bd-config-file\") pod \"limitador-limitador-64c94fd64f-9nzrt\" (UID: \"c638e86b-4d59-46fd-ad8c-49d5348889bd\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-9nzrt"
Jul 26 15:58:03.631365 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:03.631245    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdlfp\" (UniqueName: \"kubernetes.io/projected/c638e86b-4d59-46fd-ad8c-49d5348889bd-kube-api-access-tdlfp\") pod \"limitador-limitador-64c94fd64f-9nzrt\" (UID: \"c638e86b-4d59-46fd-ad8c-49d5348889bd\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-9nzrt"
Jul 26 15:58:03.631869 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:03.631846    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c638e86b-4d59-46fd-ad8c-49d5348889bd-config-file\") pod \"limitador-limitador-64c94fd64f-9nzrt\" (UID: \"c638e86b-4d59-46fd-ad8c-49d5348889bd\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-9nzrt"
Jul 26 15:58:03.638793 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:03.638775    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdlfp\" (UniqueName: \"kubernetes.io/projected/c638e86b-4d59-46fd-ad8c-49d5348889bd-kube-api-access-tdlfp\") pod \"limitador-limitador-64c94fd64f-9nzrt\" (UID: \"c638e86b-4d59-46fd-ad8c-49d5348889bd\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-9nzrt"
Jul 26 15:58:03.770660 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:03.770629    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-9nzrt"
Jul 26 15:58:03.904528 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:03.904490    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-9nzrt"]
Jul 26 15:58:03.907348 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:58:03.907322    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc638e86b_4d59_46fd_ad8c_49d5348889bd.slice/crio-55e4706cd3b395009478b1af256229bdff0adffbc7a0a1c69a678169c2a118ea WatchSource:0}: Error finding container 55e4706cd3b395009478b1af256229bdff0adffbc7a0a1c69a678169c2a118ea: Status 404 returned error can't find the container with id 55e4706cd3b395009478b1af256229bdff0adffbc7a0a1c69a678169c2a118ea
Jul 26 15:58:04.292115 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:04.292075    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:58:04.346630 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:04.346594    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:58:04.582590 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:04.582452    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-9nzrt" event={"ID":"c638e86b-4d59-46fd-ad8c-49d5348889bd","Type":"ContainerStarted","Data":"55e4706cd3b395009478b1af256229bdff0adffbc7a0a1c69a678169c2a118ea"}
Jul 26 15:58:06.593562 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:06.593538    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-9nzrt" event={"ID":"c638e86b-4d59-46fd-ad8c-49d5348889bd","Type":"ContainerStarted","Data":"3d278c09bd956185766e019c8cb8d2ab1876416c8f2ad026c4bc9f6713b4fd9a"}
Jul 26 15:58:06.593891 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:06.593669    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-9nzrt"
Jul 26 15:58:06.608252 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:06.608208    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-9nzrt" podStartSLOduration=0.990607214 podStartE2EDuration="3.608191394s" totalImagesPullingTime="2.61758418s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:58:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:58:06.607304544 +0000 UTC m=+478.711530381" watchObservedRunningTime="2026-07-26 15:58:06.608191394 +0000 UTC m=+478.712417203"
Jul 26 15:58:17.600095 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:17.600069    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-9nzrt"
Jul 26 15:58:19.661508 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:19.661475    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-9nzrt"]
Jul 26 15:58:19.661924 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:19.661728    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-9nzrt" podUID="c638e86b-4d59-46fd-ad8c-49d5348889bd" containerName="limitador" containerID="cri-o://3d278c09bd956185766e019c8cb8d2ab1876416c8f2ad026c4bc9f6713b4fd9a" gracePeriod=30
Jul 26 15:58:20.220107 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:20.220041    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-9nzrt"
Jul 26 15:58:20.281368 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:20.281338    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c638e86b-4d59-46fd-ad8c-49d5348889bd-config-file\") pod \"c638e86b-4d59-46fd-ad8c-49d5348889bd\" (UID: \"c638e86b-4d59-46fd-ad8c-49d5348889bd\") "
Jul 26 15:58:20.281529 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:20.281423    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tdlfp\" (UniqueName: \"kubernetes.io/projected/c638e86b-4d59-46fd-ad8c-49d5348889bd-kube-api-access-tdlfp\") pod \"c638e86b-4d59-46fd-ad8c-49d5348889bd\" (UID: \"c638e86b-4d59-46fd-ad8c-49d5348889bd\") "
Jul 26 15:58:20.281684 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:20.281660    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c638e86b-4d59-46fd-ad8c-49d5348889bd-config-file" (OuterVolumeSpecName: "config-file") pod "c638e86b-4d59-46fd-ad8c-49d5348889bd" (UID: "c638e86b-4d59-46fd-ad8c-49d5348889bd"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 26 15:58:20.283440 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:20.283413    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c638e86b-4d59-46fd-ad8c-49d5348889bd-kube-api-access-tdlfp" (OuterVolumeSpecName: "kube-api-access-tdlfp") pod "c638e86b-4d59-46fd-ad8c-49d5348889bd" (UID: "c638e86b-4d59-46fd-ad8c-49d5348889bd"). InnerVolumeSpecName "kube-api-access-tdlfp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 26 15:58:20.381997 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:20.381970    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdlfp\" (UniqueName: \"kubernetes.io/projected/c638e86b-4d59-46fd-ad8c-49d5348889bd-kube-api-access-tdlfp\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 15:58:20.381997 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:20.381996    2563 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c638e86b-4d59-46fd-ad8c-49d5348889bd-config-file\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 15:58:20.655574 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:20.655540    2563 generic.go:356] "Generic (PLEG): container finished" podID="c638e86b-4d59-46fd-ad8c-49d5348889bd" containerID="3d278c09bd956185766e019c8cb8d2ab1876416c8f2ad026c4bc9f6713b4fd9a" exitCode=0
Jul 26 15:58:20.655797 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:20.655584    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-9nzrt"
Jul 26 15:58:20.655797 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:20.655625    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-9nzrt" event={"ID":"c638e86b-4d59-46fd-ad8c-49d5348889bd","Type":"ContainerDied","Data":"3d278c09bd956185766e019c8cb8d2ab1876416c8f2ad026c4bc9f6713b4fd9a"}
Jul 26 15:58:20.655797 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:20.655682    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-9nzrt" event={"ID":"c638e86b-4d59-46fd-ad8c-49d5348889bd","Type":"ContainerDied","Data":"55e4706cd3b395009478b1af256229bdff0adffbc7a0a1c69a678169c2a118ea"}
Jul 26 15:58:20.655797 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:20.655711    2563 scope.go:117] "RemoveContainer" containerID="3d278c09bd956185766e019c8cb8d2ab1876416c8f2ad026c4bc9f6713b4fd9a"
Jul 26 15:58:20.664372 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:20.664154    2563 scope.go:117] "RemoveContainer" containerID="3d278c09bd956185766e019c8cb8d2ab1876416c8f2ad026c4bc9f6713b4fd9a"
Jul 26 15:58:20.664627 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:58:20.664415    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d278c09bd956185766e019c8cb8d2ab1876416c8f2ad026c4bc9f6713b4fd9a\": container with ID starting with 3d278c09bd956185766e019c8cb8d2ab1876416c8f2ad026c4bc9f6713b4fd9a not found: ID does not exist" containerID="3d278c09bd956185766e019c8cb8d2ab1876416c8f2ad026c4bc9f6713b4fd9a"
Jul 26 15:58:20.664627 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:20.664449    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d278c09bd956185766e019c8cb8d2ab1876416c8f2ad026c4bc9f6713b4fd9a"} err="failed to get container status \"3d278c09bd956185766e019c8cb8d2ab1876416c8f2ad026c4bc9f6713b4fd9a\": rpc error: code = NotFound desc = could not find container \"3d278c09bd956185766e019c8cb8d2ab1876416c8f2ad026c4bc9f6713b4fd9a\": container with ID starting with 3d278c09bd956185766e019c8cb8d2ab1876416c8f2ad026c4bc9f6713b4fd9a not found: ID does not exist"
Jul 26 15:58:20.670651 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:20.670630    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-9nzrt"]
Jul 26 15:58:20.673098 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:20.673078    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-9nzrt"]
Jul 26 15:58:22.372481 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:22.372448    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c638e86b-4d59-46fd-ad8c-49d5348889bd" path="/var/lib/kubelet/pods/c638e86b-4d59-46fd-ad8c-49d5348889bd/volumes"
Jul 26 15:58:23.071710 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:23.071661    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:58:23.196493 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:23.196456    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:58:23.540811 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:23.540782    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-g6vcc"]
Jul 26 15:58:23.541275 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:23.541258    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c638e86b-4d59-46fd-ad8c-49d5348889bd" containerName="limitador"
Jul 26 15:58:23.541338 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:23.541279    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c638e86b-4d59-46fd-ad8c-49d5348889bd" containerName="limitador"
Jul 26 15:58:23.541412 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:23.541391    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c638e86b-4d59-46fd-ad8c-49d5348889bd" containerName="limitador"
Jul 26 15:58:23.546042 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:23.546019    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-g6vcc"
Jul 26 15:58:23.548535 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:23.548512    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 26 15:58:23.548535 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:23.548532    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Jul 26 15:58:23.548703 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:23.548606    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 26 15:58:23.549268 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:23.549209    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-ljlld\""
Jul 26 15:58:23.551264 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:23.551245    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-g6vcc"]
Jul 26 15:58:23.575445 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:23.575424    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:58:23.713059 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:23.713032    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f6d59668-5579-45c0-a659-3712c99a6fe3-data\") pod \"postgres-69fb4594cf-g6vcc\" (UID: \"f6d59668-5579-45c0-a659-3712c99a6fe3\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-g6vcc"
Jul 26 15:58:23.713223 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:23.713092    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc2hz\" (UniqueName: \"kubernetes.io/projected/f6d59668-5579-45c0-a659-3712c99a6fe3-kube-api-access-rc2hz\") pod \"postgres-69fb4594cf-g6vcc\" (UID: \"f6d59668-5579-45c0-a659-3712c99a6fe3\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-g6vcc"
Jul 26 15:58:23.814521 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:23.814450    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f6d59668-5579-45c0-a659-3712c99a6fe3-data\") pod \"postgres-69fb4594cf-g6vcc\" (UID: \"f6d59668-5579-45c0-a659-3712c99a6fe3\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-g6vcc"
Jul 26 15:58:23.814642 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:23.814555    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc2hz\" (UniqueName: \"kubernetes.io/projected/f6d59668-5579-45c0-a659-3712c99a6fe3-kube-api-access-rc2hz\") pod \"postgres-69fb4594cf-g6vcc\" (UID: \"f6d59668-5579-45c0-a659-3712c99a6fe3\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-g6vcc"
Jul 26 15:58:23.814807 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:23.814792    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f6d59668-5579-45c0-a659-3712c99a6fe3-data\") pod \"postgres-69fb4594cf-g6vcc\" (UID: \"f6d59668-5579-45c0-a659-3712c99a6fe3\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-g6vcc"
Jul 26 15:58:23.822735 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:23.822713    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc2hz\" (UniqueName: \"kubernetes.io/projected/f6d59668-5579-45c0-a659-3712c99a6fe3-kube-api-access-rc2hz\") pod \"postgres-69fb4594cf-g6vcc\" (UID: \"f6d59668-5579-45c0-a659-3712c99a6fe3\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-g6vcc"
Jul 26 15:58:23.859257 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:23.859231    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-g6vcc"
Jul 26 15:58:24.000601 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:24.000576    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-g6vcc"]
Jul 26 15:58:24.002984 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:58:24.002959    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6d59668_5579_45c0_a659_3712c99a6fe3.slice/crio-b65b17d8a50daea3bc042bc608e5e6c91d13f4c2da401a8d5cdfdd7e2c51b5b2 WatchSource:0}: Error finding container b65b17d8a50daea3bc042bc608e5e6c91d13f4c2da401a8d5cdfdd7e2c51b5b2: Status 404 returned error can't find the container with id b65b17d8a50daea3bc042bc608e5e6c91d13f4c2da401a8d5cdfdd7e2c51b5b2
Jul 26 15:58:24.678582 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:24.678466    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-g6vcc" event={"ID":"f6d59668-5579-45c0-a659-3712c99a6fe3","Type":"ContainerStarted","Data":"b65b17d8a50daea3bc042bc608e5e6c91d13f4c2da401a8d5cdfdd7e2c51b5b2"}
Jul 26 15:58:29.703406 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:29.703372    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-g6vcc" event={"ID":"f6d59668-5579-45c0-a659-3712c99a6fe3","Type":"ContainerStarted","Data":"d5a0b560ced5a0c9cc8c72188254827706aa87a1c6a44db602bf313bb5660d6f"}
Jul 26 15:58:29.703822 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:29.703479    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-g6vcc"
Jul 26 15:58:29.718544 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:29.718484    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-g6vcc" podStartSLOduration=1.911605825 podStartE2EDuration="6.71846479s" totalImagesPullingTime="4.806858965s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:58:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:58:29.716918368 +0000 UTC m=+501.821144178" watchObservedRunningTime="2026-07-26 15:58:29.71846479 +0000 UTC m=+501.822690603"
Jul 26 15:58:35.736485 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:35.736456    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-g6vcc"
Jul 26 15:58:36.038021 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:36.037931    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:58:36.140187 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:36.140156    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:58:42.551527 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:42.551495    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:58:42.557493 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:42.557469    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-cdcc95985-stj6p"]
Jul 26 15:58:42.565908 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:42.565887    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-cdcc95985-stj6p"
Jul 26 15:58:42.572039 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:42.572014    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 26 15:58:42.572039 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:42.572024    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-dc6f7\""
Jul 26 15:58:42.572250 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:42.572160    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 26 15:58:42.575362 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:42.575340    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8d40642d-272b-42fe-850c-0b2b37a3909d-webhook-cert\") pod \"maas-controller-cdcc95985-stj6p\" (UID: \"8d40642d-272b-42fe-850c-0b2b37a3909d\") " pod="opendatahub/maas-controller-cdcc95985-stj6p"
Jul 26 15:58:42.575472 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:42.575389    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2j82\" (UniqueName: \"kubernetes.io/projected/8d40642d-272b-42fe-850c-0b2b37a3909d-kube-api-access-f2j82\") pod \"maas-controller-cdcc95985-stj6p\" (UID: \"8d40642d-272b-42fe-850c-0b2b37a3909d\") " pod="opendatahub/maas-controller-cdcc95985-stj6p"
Jul 26 15:58:42.578737 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:42.578715    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-cdcc95985-stj6p"]
Jul 26 15:58:42.648481 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:42.648451    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:58:42.676117 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:42.676091    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8d40642d-272b-42fe-850c-0b2b37a3909d-webhook-cert\") pod \"maas-controller-cdcc95985-stj6p\" (UID: \"8d40642d-272b-42fe-850c-0b2b37a3909d\") " pod="opendatahub/maas-controller-cdcc95985-stj6p"
Jul 26 15:58:42.676224 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:42.676134    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2j82\" (UniqueName: \"kubernetes.io/projected/8d40642d-272b-42fe-850c-0b2b37a3909d-kube-api-access-f2j82\") pod \"maas-controller-cdcc95985-stj6p\" (UID: \"8d40642d-272b-42fe-850c-0b2b37a3909d\") " pod="opendatahub/maas-controller-cdcc95985-stj6p"
Jul 26 15:58:42.676267 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:58:42.676242    2563 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Jul 26 15:58:42.676315 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:58:42.676305    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d40642d-272b-42fe-850c-0b2b37a3909d-webhook-cert podName:8d40642d-272b-42fe-850c-0b2b37a3909d nodeName:}" failed. No retries permitted until 2026-07-26 15:58:43.176288171 +0000 UTC m=+515.280513957 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/8d40642d-272b-42fe-850c-0b2b37a3909d-webhook-cert") pod "maas-controller-cdcc95985-stj6p" (UID: "8d40642d-272b-42fe-850c-0b2b37a3909d") : secret "maas-controller-webhook-cert" not found
Jul 26 15:58:42.684722 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:42.684690    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2j82\" (UniqueName: \"kubernetes.io/projected/8d40642d-272b-42fe-850c-0b2b37a3909d-kube-api-access-f2j82\") pod \"maas-controller-cdcc95985-stj6p\" (UID: \"8d40642d-272b-42fe-850c-0b2b37a3909d\") " pod="opendatahub/maas-controller-cdcc95985-stj6p"
Jul 26 15:58:42.746859 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:42.746824    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:58:42.750234 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:42.750206    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5fb95c8f7b-2xbnv"]
Jul 26 15:58:42.753968 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:42.753945    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fb95c8f7b-2xbnv"
Jul 26 15:58:42.763807 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:42.763782    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5fb95c8f7b-2xbnv"]
Jul 26 15:58:42.776875 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:42.776849    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2f11f601-c407-4354-874b-8a83a50b725d-webhook-cert\") pod \"maas-controller-5fb95c8f7b-2xbnv\" (UID: \"2f11f601-c407-4354-874b-8a83a50b725d\") " pod="opendatahub/maas-controller-5fb95c8f7b-2xbnv"
Jul 26 15:58:42.777014 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:42.776920    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hqr7\" (UniqueName: \"kubernetes.io/projected/2f11f601-c407-4354-874b-8a83a50b725d-kube-api-access-8hqr7\") pod \"maas-controller-5fb95c8f7b-2xbnv\" (UID: \"2f11f601-c407-4354-874b-8a83a50b725d\") " pod="opendatahub/maas-controller-5fb95c8f7b-2xbnv"
Jul 26 15:58:42.877986 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:42.877900    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2f11f601-c407-4354-874b-8a83a50b725d-webhook-cert\") pod \"maas-controller-5fb95c8f7b-2xbnv\" (UID: \"2f11f601-c407-4354-874b-8a83a50b725d\") " pod="opendatahub/maas-controller-5fb95c8f7b-2xbnv"
Jul 26 15:58:42.877986 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:42.877967    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hqr7\" (UniqueName: \"kubernetes.io/projected/2f11f601-c407-4354-874b-8a83a50b725d-kube-api-access-8hqr7\") pod \"maas-controller-5fb95c8f7b-2xbnv\" (UID: \"2f11f601-c407-4354-874b-8a83a50b725d\") " pod="opendatahub/maas-controller-5fb95c8f7b-2xbnv"
Jul 26 15:58:42.878215 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:58:42.878066    2563 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Jul 26 15:58:42.878215 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:58:42.878139    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f11f601-c407-4354-874b-8a83a50b725d-webhook-cert podName:2f11f601-c407-4354-874b-8a83a50b725d nodeName:}" failed. No retries permitted until 2026-07-26 15:58:43.378120279 +0000 UTC m=+515.482346074 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/2f11f601-c407-4354-874b-8a83a50b725d-webhook-cert") pod "maas-controller-5fb95c8f7b-2xbnv" (UID: "2f11f601-c407-4354-874b-8a83a50b725d") : secret "maas-controller-webhook-cert" not found
Jul 26 15:58:42.888388 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:42.888366    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hqr7\" (UniqueName: \"kubernetes.io/projected/2f11f601-c407-4354-874b-8a83a50b725d-kube-api-access-8hqr7\") pod \"maas-controller-5fb95c8f7b-2xbnv\" (UID: \"2f11f601-c407-4354-874b-8a83a50b725d\") " pod="opendatahub/maas-controller-5fb95c8f7b-2xbnv"
Jul 26 15:58:43.096256 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:43.096220    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:58:43.180257 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:43.180170    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8d40642d-272b-42fe-850c-0b2b37a3909d-webhook-cert\") pod \"maas-controller-cdcc95985-stj6p\" (UID: \"8d40642d-272b-42fe-850c-0b2b37a3909d\") " pod="opendatahub/maas-controller-cdcc95985-stj6p"
Jul 26 15:58:43.182593 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:43.182569    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8d40642d-272b-42fe-850c-0b2b37a3909d-webhook-cert\") pod \"maas-controller-cdcc95985-stj6p\" (UID: \"8d40642d-272b-42fe-850c-0b2b37a3909d\") " pod="opendatahub/maas-controller-cdcc95985-stj6p"
Jul 26 15:58:43.382454 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:43.382418    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2f11f601-c407-4354-874b-8a83a50b725d-webhook-cert\") pod \"maas-controller-5fb95c8f7b-2xbnv\" (UID: \"2f11f601-c407-4354-874b-8a83a50b725d\") " pod="opendatahub/maas-controller-5fb95c8f7b-2xbnv"
Jul 26 15:58:43.384912 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:43.384887    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2f11f601-c407-4354-874b-8a83a50b725d-webhook-cert\") pod \"maas-controller-5fb95c8f7b-2xbnv\" (UID: \"2f11f601-c407-4354-874b-8a83a50b725d\") " pod="opendatahub/maas-controller-5fb95c8f7b-2xbnv"
Jul 26 15:58:43.476125 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:43.476094    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-cdcc95985-stj6p"
Jul 26 15:58:43.611940 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:43.608442    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-cdcc95985-stj6p"]
Jul 26 15:58:43.665939 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:43.665901    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fb95c8f7b-2xbnv"
Jul 26 15:58:43.770050 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:43.770015    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-cdcc95985-stj6p" event={"ID":"8d40642d-272b-42fe-850c-0b2b37a3909d","Type":"ContainerStarted","Data":"e1dd15945ad3882eef631e95187e8decd1b67f547dc8f17a9678c3023714cded"}
Jul 26 15:58:43.793660 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:43.793636    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5fb95c8f7b-2xbnv"]
Jul 26 15:58:43.795855 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:58:43.795820    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f11f601_c407_4354_874b_8a83a50b725d.slice/crio-65c5179fcd3f44cb41935c825e51773aaaf90a246ede0907cc8fa76e53fc45fe WatchSource:0}: Error finding container 65c5179fcd3f44cb41935c825e51773aaaf90a246ede0907cc8fa76e53fc45fe: Status 404 returned error can't find the container with id 65c5179fcd3f44cb41935c825e51773aaaf90a246ede0907cc8fa76e53fc45fe
Jul 26 15:58:44.785235 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:44.785192    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fb95c8f7b-2xbnv" event={"ID":"2f11f601-c407-4354-874b-8a83a50b725d","Type":"ContainerStarted","Data":"65c5179fcd3f44cb41935c825e51773aaaf90a246ede0907cc8fa76e53fc45fe"}
Jul 26 15:58:46.272145 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:46.272114    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:58:46.372903 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:46.372825    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:58:46.796938 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:46.796904    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-cdcc95985-stj6p" event={"ID":"8d40642d-272b-42fe-850c-0b2b37a3909d","Type":"ContainerStarted","Data":"b5c32bc84941120f47cfe3aacd74edbb2e481d19d1439e8639c36aaff735ddf8"}
Jul 26 15:58:46.797132 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:46.796974    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-cdcc95985-stj6p"
Jul 26 15:58:46.798539 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:46.798511    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fb95c8f7b-2xbnv" event={"ID":"2f11f601-c407-4354-874b-8a83a50b725d","Type":"ContainerStarted","Data":"780d503446aa2ae674c2eb66b9c391c34ea7c42d0c922f625217fb9e4ace0407"}
Jul 26 15:58:46.798636 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:46.798574    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5fb95c8f7b-2xbnv"
Jul 26 15:58:46.816700 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:46.816659    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-cdcc95985-stj6p" podStartSLOduration=2.326147548 podStartE2EDuration="4.816644081s" totalImagesPullingTime="2.490496533s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:58:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:58:46.814654163 +0000 UTC m=+518.918879974" watchObservedRunningTime="2026-07-26 15:58:46.816644081 +0000 UTC m=+518.920869889"
Jul 26 15:58:46.833948 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:46.833897    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5fb95c8f7b-2xbnv" podStartSLOduration=2.5215892159999997 podStartE2EDuration="4.833883231s" totalImagesPullingTime="2.312294015s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:58:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:58:46.832381101 +0000 UTC m=+518.936606911" watchObservedRunningTime="2026-07-26 15:58:46.833883231 +0000 UTC m=+518.938109040"
Jul 26 15:58:57.805519 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:57.805488    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5fb95c8f7b-2xbnv"
Jul 26 15:58:57.806023 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:57.805569    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-cdcc95985-stj6p"
Jul 26 15:58:57.863310 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:57.863274    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-cdcc95985-stj6p"]
Jul 26 15:58:57.863472 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:57.863448    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-cdcc95985-stj6p" podUID="8d40642d-272b-42fe-850c-0b2b37a3909d" containerName="manager" containerID="cri-o://b5c32bc84941120f47cfe3aacd74edbb2e481d19d1439e8639c36aaff735ddf8" gracePeriod=30
Jul 26 15:58:58.106214 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:58.106191    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-cdcc95985-stj6p"
Jul 26 15:58:58.214924 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:58.214894    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8d40642d-272b-42fe-850c-0b2b37a3909d-webhook-cert\") pod \"8d40642d-272b-42fe-850c-0b2b37a3909d\" (UID: \"8d40642d-272b-42fe-850c-0b2b37a3909d\") "
Jul 26 15:58:58.215119 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:58.215018    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f2j82\" (UniqueName: \"kubernetes.io/projected/8d40642d-272b-42fe-850c-0b2b37a3909d-kube-api-access-f2j82\") pod \"8d40642d-272b-42fe-850c-0b2b37a3909d\" (UID: \"8d40642d-272b-42fe-850c-0b2b37a3909d\") "
Jul 26 15:58:58.217017 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:58.216989    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d40642d-272b-42fe-850c-0b2b37a3909d-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "8d40642d-272b-42fe-850c-0b2b37a3909d" (UID: "8d40642d-272b-42fe-850c-0b2b37a3909d"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 26 15:58:58.217135 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:58.217065    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d40642d-272b-42fe-850c-0b2b37a3909d-kube-api-access-f2j82" (OuterVolumeSpecName: "kube-api-access-f2j82") pod "8d40642d-272b-42fe-850c-0b2b37a3909d" (UID: "8d40642d-272b-42fe-850c-0b2b37a3909d"). InnerVolumeSpecName "kube-api-access-f2j82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 26 15:58:58.316487 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:58.316437    2563 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8d40642d-272b-42fe-850c-0b2b37a3909d-webhook-cert\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 15:58:58.316487 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:58.316484    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2j82\" (UniqueName: \"kubernetes.io/projected/8d40642d-272b-42fe-850c-0b2b37a3909d-kube-api-access-f2j82\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 15:58:58.859211 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:58.859176    2563 generic.go:356] "Generic (PLEG): container finished" podID="8d40642d-272b-42fe-850c-0b2b37a3909d" containerID="b5c32bc84941120f47cfe3aacd74edbb2e481d19d1439e8639c36aaff735ddf8" exitCode=0
Jul 26 15:58:58.859686 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:58.859236    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-cdcc95985-stj6p"
Jul 26 15:58:58.859686 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:58.859267    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-cdcc95985-stj6p" event={"ID":"8d40642d-272b-42fe-850c-0b2b37a3909d","Type":"ContainerDied","Data":"b5c32bc84941120f47cfe3aacd74edbb2e481d19d1439e8639c36aaff735ddf8"}
Jul 26 15:58:58.859686 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:58.859329    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-cdcc95985-stj6p" event={"ID":"8d40642d-272b-42fe-850c-0b2b37a3909d","Type":"ContainerDied","Data":"e1dd15945ad3882eef631e95187e8decd1b67f547dc8f17a9678c3023714cded"}
Jul 26 15:58:58.859686 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:58.859363    2563 scope.go:117] "RemoveContainer" containerID="b5c32bc84941120f47cfe3aacd74edbb2e481d19d1439e8639c36aaff735ddf8"
Jul 26 15:58:58.867783 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:58.867566    2563 scope.go:117] "RemoveContainer" containerID="b5c32bc84941120f47cfe3aacd74edbb2e481d19d1439e8639c36aaff735ddf8"
Jul 26 15:58:58.867864 ip-10-0-133-204 kubenswrapper[2563]: E0726 15:58:58.867841    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5c32bc84941120f47cfe3aacd74edbb2e481d19d1439e8639c36aaff735ddf8\": container with ID starting with b5c32bc84941120f47cfe3aacd74edbb2e481d19d1439e8639c36aaff735ddf8 not found: ID does not exist" containerID="b5c32bc84941120f47cfe3aacd74edbb2e481d19d1439e8639c36aaff735ddf8"
Jul 26 15:58:58.867903 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:58.867873    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5c32bc84941120f47cfe3aacd74edbb2e481d19d1439e8639c36aaff735ddf8"} err="failed to get container status \"b5c32bc84941120f47cfe3aacd74edbb2e481d19d1439e8639c36aaff735ddf8\": rpc error: code = NotFound desc = could not find container \"b5c32bc84941120f47cfe3aacd74edbb2e481d19d1439e8639c36aaff735ddf8\": container with ID starting with b5c32bc84941120f47cfe3aacd74edbb2e481d19d1439e8639c36aaff735ddf8 not found: ID does not exist"
Jul 26 15:58:58.874868 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:58.874846    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-cdcc95985-stj6p"]
Jul 26 15:58:58.879153 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:58:58.879133    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-cdcc95985-stj6p"]
Jul 26 15:59:00.372793 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:00.372699    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d40642d-272b-42fe-850c-0b2b37a3909d" path="/var/lib/kubelet/pods/8d40642d-272b-42fe-850c-0b2b37a3909d/volumes"
Jul 26 15:59:17.971834 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:17.971796    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:59:18.069879 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:18.069844    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:59:18.481885 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:18.481846    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:59:18.542155 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:18.542119    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-849dbfd774-dj2wk"]
Jul 26 15:59:18.542473 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:18.542456    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d40642d-272b-42fe-850c-0b2b37a3909d" containerName="manager"
Jul 26 15:59:18.542473 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:18.542471    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d40642d-272b-42fe-850c-0b2b37a3909d" containerName="manager"
Jul 26 15:59:18.542654 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:18.542543    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d40642d-272b-42fe-850c-0b2b37a3909d" containerName="manager"
Jul 26 15:59:18.562464 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:18.562436    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-849dbfd774-dj2wk"]
Jul 26 15:59:18.562616 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:18.562540    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-849dbfd774-dj2wk"
Jul 26 15:59:18.564878 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:18.564854    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Jul 26 15:59:18.565006 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:18.564926    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-dgfss\""
Jul 26 15:59:18.699620 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:18.699590    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08a0697f-7477-4170-9e68-f32144f104b4-openshift-service-ca\") pod \"maas-api-849dbfd774-dj2wk\" (UID: \"08a0697f-7477-4170-9e68-f32144f104b4\") " pod="odh-ai-gateway-infra/maas-api-849dbfd774-dj2wk"
Jul 26 15:59:18.699792 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:18.699629    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btc2n\" (UniqueName: \"kubernetes.io/projected/08a0697f-7477-4170-9e68-f32144f104b4-kube-api-access-btc2n\") pod \"maas-api-849dbfd774-dj2wk\" (UID: \"08a0697f-7477-4170-9e68-f32144f104b4\") " pod="odh-ai-gateway-infra/maas-api-849dbfd774-dj2wk"
Jul 26 15:59:18.699792 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:18.699686    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08a0697f-7477-4170-9e68-f32144f104b4-maas-api-tls\") pod \"maas-api-849dbfd774-dj2wk\" (UID: \"08a0697f-7477-4170-9e68-f32144f104b4\") " pod="odh-ai-gateway-infra/maas-api-849dbfd774-dj2wk"
Jul 26 15:59:18.716603 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:18.716570    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:59:18.800400 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:18.800306    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08a0697f-7477-4170-9e68-f32144f104b4-openshift-service-ca\") pod \"maas-api-849dbfd774-dj2wk\" (UID: \"08a0697f-7477-4170-9e68-f32144f104b4\") " pod="odh-ai-gateway-infra/maas-api-849dbfd774-dj2wk"
Jul 26 15:59:18.800400 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:18.800364    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btc2n\" (UniqueName: \"kubernetes.io/projected/08a0697f-7477-4170-9e68-f32144f104b4-kube-api-access-btc2n\") pod \"maas-api-849dbfd774-dj2wk\" (UID: \"08a0697f-7477-4170-9e68-f32144f104b4\") " pod="odh-ai-gateway-infra/maas-api-849dbfd774-dj2wk"
Jul 26 15:59:18.800675 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:18.800418    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08a0697f-7477-4170-9e68-f32144f104b4-maas-api-tls\") pod \"maas-api-849dbfd774-dj2wk\" (UID: \"08a0697f-7477-4170-9e68-f32144f104b4\") " pod="odh-ai-gateway-infra/maas-api-849dbfd774-dj2wk"
Jul 26 15:59:18.801076 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:18.801053    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08a0697f-7477-4170-9e68-f32144f104b4-openshift-service-ca\") pod \"maas-api-849dbfd774-dj2wk\" (UID: \"08a0697f-7477-4170-9e68-f32144f104b4\") " pod="odh-ai-gateway-infra/maas-api-849dbfd774-dj2wk"
Jul 26 15:59:18.803046 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:18.803026    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08a0697f-7477-4170-9e68-f32144f104b4-maas-api-tls\") pod \"maas-api-849dbfd774-dj2wk\" (UID: \"08a0697f-7477-4170-9e68-f32144f104b4\") " pod="odh-ai-gateway-infra/maas-api-849dbfd774-dj2wk"
Jul 26 15:59:18.809562 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:18.809536    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btc2n\" (UniqueName: \"kubernetes.io/projected/08a0697f-7477-4170-9e68-f32144f104b4-kube-api-access-btc2n\") pod \"maas-api-849dbfd774-dj2wk\" (UID: \"08a0697f-7477-4170-9e68-f32144f104b4\") " pod="odh-ai-gateway-infra/maas-api-849dbfd774-dj2wk"
Jul 26 15:59:18.873014 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:18.872979    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-849dbfd774-dj2wk"
Jul 26 15:59:19.020705 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:19.020648    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-849dbfd774-dj2wk"]
Jul 26 15:59:19.023277 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:59:19.023251    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08a0697f_7477_4170_9e68_f32144f104b4.slice/crio-7af64baaa369c8721f1f059d033622954102d16bd4578703f35dea30f0267735 WatchSource:0}: Error finding container 7af64baaa369c8721f1f059d033622954102d16bd4578703f35dea30f0267735: Status 404 returned error can't find the container with id 7af64baaa369c8721f1f059d033622954102d16bd4578703f35dea30f0267735
Jul 26 15:59:19.061675 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:19.061589    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:59:19.157407 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:19.157377    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:59:19.164902 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:19.164872    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-dhspm"]
Jul 26 15:59:19.169215 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:19.169196    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-dhspm"
Jul 26 15:59:19.171430 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:19.171400    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 26 15:59:19.171576 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:19.171456    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-2mvq5\""
Jul 26 15:59:19.180524 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:19.180505    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-dhspm"]
Jul 26 15:59:19.182003 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:19.181983    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-rc9pz"]
Jul 26 15:59:19.185271 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:19.185253    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-rc9pz"
Jul 26 15:59:19.197472 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:19.197443    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-rc9pz"]
Jul 26 15:59:19.305809 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:19.305743    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c08ac5a-85ff-4504-9d77-b8f41be0242a-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-rc9pz\" (UID: \"9c08ac5a-85ff-4504-9d77-b8f41be0242a\") " pod="openshift-ingress/payload-processing-5cc6cd444b-rc9pz"
Jul 26 15:59:19.306000 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:19.305913    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh9m8\" (UniqueName: \"kubernetes.io/projected/9c08ac5a-85ff-4504-9d77-b8f41be0242a-kube-api-access-wh9m8\") pod \"payload-processing-5cc6cd444b-rc9pz\" (UID: \"9c08ac5a-85ff-4504-9d77-b8f41be0242a\") " pod="openshift-ingress/payload-processing-5cc6cd444b-rc9pz"
Jul 26 15:59:19.306067 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:19.306004    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0863e78-2d3f-482a-bc97-6738e35a6618-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-dhspm\" (UID: \"f0863e78-2d3f-482a-bc97-6738e35a6618\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-dhspm"
Jul 26 15:59:19.306130 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:19.306090    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gs2rw\" (UniqueName: \"kubernetes.io/projected/f0863e78-2d3f-482a-bc97-6738e35a6618-kube-api-access-gs2rw\") pod \"payload-pre-processing-687dbb6cf6-dhspm\" (UID: \"f0863e78-2d3f-482a-bc97-6738e35a6618\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-dhspm"
Jul 26 15:59:19.407246 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:19.407176    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh9m8\" (UniqueName: \"kubernetes.io/projected/9c08ac5a-85ff-4504-9d77-b8f41be0242a-kube-api-access-wh9m8\") pod \"payload-processing-5cc6cd444b-rc9pz\" (UID: \"9c08ac5a-85ff-4504-9d77-b8f41be0242a\") " pod="openshift-ingress/payload-processing-5cc6cd444b-rc9pz"
Jul 26 15:59:19.407246 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:19.407234    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0863e78-2d3f-482a-bc97-6738e35a6618-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-dhspm\" (UID: \"f0863e78-2d3f-482a-bc97-6738e35a6618\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-dhspm"
Jul 26 15:59:19.407450 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:19.407267    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs2rw\" (UniqueName: \"kubernetes.io/projected/f0863e78-2d3f-482a-bc97-6738e35a6618-kube-api-access-gs2rw\") pod \"payload-pre-processing-687dbb6cf6-dhspm\" (UID: \"f0863e78-2d3f-482a-bc97-6738e35a6618\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-dhspm"
Jul 26 15:59:19.407450 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:19.407286    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c08ac5a-85ff-4504-9d77-b8f41be0242a-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-rc9pz\" (UID: \"9c08ac5a-85ff-4504-9d77-b8f41be0242a\") " pod="openshift-ingress/payload-processing-5cc6cd444b-rc9pz"
Jul 26 15:59:19.407951 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:19.407931    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0863e78-2d3f-482a-bc97-6738e35a6618-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-dhspm\" (UID: \"f0863e78-2d3f-482a-bc97-6738e35a6618\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-dhspm"
Jul 26 15:59:19.408008 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:19.407957    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c08ac5a-85ff-4504-9d77-b8f41be0242a-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-rc9pz\" (UID: \"9c08ac5a-85ff-4504-9d77-b8f41be0242a\") " pod="openshift-ingress/payload-processing-5cc6cd444b-rc9pz"
Jul 26 15:59:19.414724 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:19.414702    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs2rw\" (UniqueName: \"kubernetes.io/projected/f0863e78-2d3f-482a-bc97-6738e35a6618-kube-api-access-gs2rw\") pod \"payload-pre-processing-687dbb6cf6-dhspm\" (UID: \"f0863e78-2d3f-482a-bc97-6738e35a6618\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-dhspm"
Jul 26 15:59:19.414852 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:19.414750    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh9m8\" (UniqueName: \"kubernetes.io/projected/9c08ac5a-85ff-4504-9d77-b8f41be0242a-kube-api-access-wh9m8\") pod \"payload-processing-5cc6cd444b-rc9pz\" (UID: \"9c08ac5a-85ff-4504-9d77-b8f41be0242a\") " pod="openshift-ingress/payload-processing-5cc6cd444b-rc9pz"
Jul 26 15:59:19.480571 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:19.480537    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-dhspm"
Jul 26 15:59:19.496755 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:19.496725    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-rc9pz"
Jul 26 15:59:19.634242 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:19.634204    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-dhspm"]
Jul 26 15:59:19.637901 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:59:19.637860    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0863e78_2d3f_482a_bc97_6738e35a6618.slice/crio-d3e6b6dc3ebd8acf8a0931d85b7df10d4f7268ce96effbacb2ed968d3214a538 WatchSource:0}: Error finding container d3e6b6dc3ebd8acf8a0931d85b7df10d4f7268ce96effbacb2ed968d3214a538: Status 404 returned error can't find the container with id d3e6b6dc3ebd8acf8a0931d85b7df10d4f7268ce96effbacb2ed968d3214a538
Jul 26 15:59:19.672568 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:19.672479    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-rc9pz"]
Jul 26 15:59:19.675451 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:59:19.675421    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c08ac5a_85ff_4504_9d77_b8f41be0242a.slice/crio-465508ffb7a10a8540877138bbcf0605cf865656c9e6907469dbc82cbbe2fa4a WatchSource:0}: Error finding container 465508ffb7a10a8540877138bbcf0605cf865656c9e6907469dbc82cbbe2fa4a: Status 404 returned error can't find the container with id 465508ffb7a10a8540877138bbcf0605cf865656c9e6907469dbc82cbbe2fa4a
Jul 26 15:59:19.961494 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:19.961410    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-dhspm" event={"ID":"f0863e78-2d3f-482a-bc97-6738e35a6618","Type":"ContainerStarted","Data":"d3e6b6dc3ebd8acf8a0931d85b7df10d4f7268ce96effbacb2ed968d3214a538"}
Jul 26 15:59:19.962664 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:19.962630    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-849dbfd774-dj2wk" event={"ID":"08a0697f-7477-4170-9e68-f32144f104b4","Type":"ContainerStarted","Data":"7af64baaa369c8721f1f059d033622954102d16bd4578703f35dea30f0267735"}
Jul 26 15:59:19.963795 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:19.963775    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-rc9pz" event={"ID":"9c08ac5a-85ff-4504-9d77-b8f41be0242a","Type":"ContainerStarted","Data":"465508ffb7a10a8540877138bbcf0605cf865656c9e6907469dbc82cbbe2fa4a"}
Jul 26 15:59:21.981167 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:21.981121    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-849dbfd774-dj2wk" event={"ID":"08a0697f-7477-4170-9e68-f32144f104b4","Type":"ContainerStarted","Data":"f61a90ad48701cd3b28ae2999039ed3f55449a30058c8a1ca314e62211e7ca8f"}
Jul 26 15:59:21.981668 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:21.981230    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-849dbfd774-dj2wk"
Jul 26 15:59:21.995969 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:21.995898    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-849dbfd774-dj2wk" podStartSLOduration=2.128463005 podStartE2EDuration="3.995876825s" totalImagesPullingTime="1.86741382s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:59:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:59:21.995148872 +0000 UTC m=+554.099374693" watchObservedRunningTime="2026-07-26 15:59:21.995876825 +0000 UTC m=+554.100102632"
Jul 26 15:59:22.987335 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:22.987298    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-rc9pz" event={"ID":"9c08ac5a-85ff-4504-9d77-b8f41be0242a","Type":"ContainerStarted","Data":"2dee9d6c0de977ff5b1eb271176548b6d96df98170327d2ad83ab830d4fe96fc"}
Jul 26 15:59:22.987816 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:22.987392    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-rc9pz"
Jul 26 15:59:22.989009 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:22.988985    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-dhspm" event={"ID":"f0863e78-2d3f-482a-bc97-6738e35a6618","Type":"ContainerStarted","Data":"a7edcdf821700e43dd537986b42b68aaee058f8528f03a46e96a7daaec9df649"}
Jul 26 15:59:23.001702 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:23.001646    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-rc9pz" podStartSLOduration=1.490398572 podStartE2EDuration="4.001627881s" totalImagesPullingTime="2.511229309s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:59:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:59:23.001017728 +0000 UTC m=+555.105243538" watchObservedRunningTime="2026-07-26 15:59:23.001627881 +0000 UTC m=+555.105853692"
Jul 26 15:59:23.015913 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:23.015852    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-dhspm" podStartSLOduration=1.474686702 podStartE2EDuration="4.015834329s" totalImagesPullingTime="2.541147627s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:59:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:59:23.015117831 +0000 UTC m=+555.119343639" watchObservedRunningTime="2026-07-26 15:59:23.015834329 +0000 UTC m=+555.120060138"
Jul 26 15:59:23.995035 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:23.995001    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-dhspm"
Jul 26 15:59:27.994980 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:27.994947    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-849dbfd774-dj2wk"
Jul 26 15:59:29.574929 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:29.574899    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:59:29.666397 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:29.666362    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:59:33.997149 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:33.997118    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-rc9pz"
Jul 26 15:59:35.001426 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:35.001389    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-dhspm"
Jul 26 15:59:45.552494 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:45.552462    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mlpjn"]
Jul 26 15:59:45.556790 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:45.556749    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mlpjn"
Jul 26 15:59:45.560227 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:45.560206    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-8278j\""
Jul 26 15:59:45.560358 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:45.560212    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 26 15:59:45.560358 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:45.560212    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Jul 26 15:59:45.560358 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:45.560212    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 26 15:59:45.565453 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:45.565202    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mlpjn"]
Jul 26 15:59:45.572710 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:45.572687    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:59:45.635182 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:45.635146    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8862d5e5-d805-49f7-ac37-e6572367feac-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mlpjn\" (UID: \"8862d5e5-d805-49f7-ac37-e6572367feac\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mlpjn"
Jul 26 15:59:45.635374 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:45.635203    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8862d5e5-d805-49f7-ac37-e6572367feac-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mlpjn\" (UID: \"8862d5e5-d805-49f7-ac37-e6572367feac\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mlpjn"
Jul 26 15:59:45.635374 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:45.635259    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8862d5e5-d805-49f7-ac37-e6572367feac-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mlpjn\" (UID: \"8862d5e5-d805-49f7-ac37-e6572367feac\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mlpjn"
Jul 26 15:59:45.635374 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:45.635282    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8862d5e5-d805-49f7-ac37-e6572367feac-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mlpjn\" (UID: \"8862d5e5-d805-49f7-ac37-e6572367feac\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mlpjn"
Jul 26 15:59:45.635509 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:45.635378    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw9hm\" (UniqueName: \"kubernetes.io/projected/8862d5e5-d805-49f7-ac37-e6572367feac-kube-api-access-vw9hm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mlpjn\" (UID: \"8862d5e5-d805-49f7-ac37-e6572367feac\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mlpjn"
Jul 26 15:59:45.635509 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:45.635434    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8862d5e5-d805-49f7-ac37-e6572367feac-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mlpjn\" (UID: \"8862d5e5-d805-49f7-ac37-e6572367feac\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mlpjn"
Jul 26 15:59:45.736255 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:45.736227    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw9hm\" (UniqueName: \"kubernetes.io/projected/8862d5e5-d805-49f7-ac37-e6572367feac-kube-api-access-vw9hm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mlpjn\" (UID: \"8862d5e5-d805-49f7-ac37-e6572367feac\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mlpjn"
Jul 26 15:59:45.736410 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:45.736272    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8862d5e5-d805-49f7-ac37-e6572367feac-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mlpjn\" (UID: \"8862d5e5-d805-49f7-ac37-e6572367feac\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mlpjn"
Jul 26 15:59:45.736410 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:45.736306    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8862d5e5-d805-49f7-ac37-e6572367feac-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mlpjn\" (UID: \"8862d5e5-d805-49f7-ac37-e6572367feac\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mlpjn"
Jul 26 15:59:45.736410 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:45.736352    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8862d5e5-d805-49f7-ac37-e6572367feac-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mlpjn\" (UID: \"8862d5e5-d805-49f7-ac37-e6572367feac\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mlpjn"
Jul 26 15:59:45.736410 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:45.736382    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8862d5e5-d805-49f7-ac37-e6572367feac-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mlpjn\" (UID: \"8862d5e5-d805-49f7-ac37-e6572367feac\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mlpjn"
Jul 26 15:59:45.736410 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:45.736408    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8862d5e5-d805-49f7-ac37-e6572367feac-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mlpjn\" (UID: \"8862d5e5-d805-49f7-ac37-e6572367feac\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mlpjn"
Jul 26 15:59:45.736709 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:45.736688    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8862d5e5-d805-49f7-ac37-e6572367feac-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mlpjn\" (UID: \"8862d5e5-d805-49f7-ac37-e6572367feac\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mlpjn"
Jul 26 15:59:45.736862 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:45.736844    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8862d5e5-d805-49f7-ac37-e6572367feac-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mlpjn\" (UID: \"8862d5e5-d805-49f7-ac37-e6572367feac\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mlpjn"
Jul 26 15:59:45.736926 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:45.736859    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8862d5e5-d805-49f7-ac37-e6572367feac-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mlpjn\" (UID: \"8862d5e5-d805-49f7-ac37-e6572367feac\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mlpjn"
Jul 26 15:59:45.738517 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:45.738495    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8862d5e5-d805-49f7-ac37-e6572367feac-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mlpjn\" (UID: \"8862d5e5-d805-49f7-ac37-e6572367feac\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mlpjn"
Jul 26 15:59:45.738753 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:45.738736    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8862d5e5-d805-49f7-ac37-e6572367feac-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mlpjn\" (UID: \"8862d5e5-d805-49f7-ac37-e6572367feac\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mlpjn"
Jul 26 15:59:45.744516 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:45.744499    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw9hm\" (UniqueName: \"kubernetes.io/projected/8862d5e5-d805-49f7-ac37-e6572367feac-kube-api-access-vw9hm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mlpjn\" (UID: \"8862d5e5-d805-49f7-ac37-e6572367feac\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mlpjn"
Jul 26 15:59:45.870670 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:45.870585    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mlpjn"
Jul 26 15:59:46.006209 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:46.006187    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mlpjn"]
Jul 26 15:59:46.008272 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:59:46.008246    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8862d5e5_d805_49f7_ac37_e6572367feac.slice/crio-eea0b69007ed5ad5755a6616977437a84bc06e8df16bd620e941a3beb6b285f8 WatchSource:0}: Error finding container eea0b69007ed5ad5755a6616977437a84bc06e8df16bd620e941a3beb6b285f8: Status 404 returned error can't find the container with id eea0b69007ed5ad5755a6616977437a84bc06e8df16bd620e941a3beb6b285f8
Jul 26 15:59:46.104646 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:46.104613    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mlpjn" event={"ID":"8862d5e5-d805-49f7-ac37-e6572367feac","Type":"ContainerStarted","Data":"eea0b69007ed5ad5755a6616977437a84bc06e8df16bd620e941a3beb6b285f8"}
Jul 26 15:59:48.115997 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:48.115960    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mlpjn" event={"ID":"8862d5e5-d805-49f7-ac37-e6572367feac","Type":"ContainerStarted","Data":"a274cf521801dc7bd13cd9dfcd21b2efa9ac3e31f5d19c54eff61fda87888cb5"}
Jul 26 15:59:48.132101 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:48.132051    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mlpjn" podStartSLOduration=2.086963791 podStartE2EDuration="3.132034564s" totalImagesPullingTime="1.045070773s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:59:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:59:48.130864398 +0000 UTC m=+580.235090208" watchObservedRunningTime="2026-07-26 15:59:48.132034564 +0000 UTC m=+580.236260372"
Jul 26 15:59:53.849955 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:53.849921    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-kvcvx"]
Jul 26 15:59:53.853637 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:53.853617    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kvcvx"
Jul 26 15:59:53.856070 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:53.856050    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Jul 26 15:59:53.862449 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:53.862425    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-kvcvx"]
Jul 26 15:59:53.869385 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:53.869353    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 15:59:53.918093 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:53.917965    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cde8a1d7-d445-4d0b-ba25-dc3e56d6657b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kvcvx\" (UID: \"cde8a1d7-d445-4d0b-ba25-dc3e56d6657b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kvcvx"
Jul 26 15:59:53.918808 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:53.918360    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zds82\" (UniqueName: \"kubernetes.io/projected/cde8a1d7-d445-4d0b-ba25-dc3e56d6657b-kube-api-access-zds82\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kvcvx\" (UID: \"cde8a1d7-d445-4d0b-ba25-dc3e56d6657b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kvcvx"
Jul 26 15:59:53.918808 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:53.918407    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cde8a1d7-d445-4d0b-ba25-dc3e56d6657b-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kvcvx\" (UID: \"cde8a1d7-d445-4d0b-ba25-dc3e56d6657b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kvcvx"
Jul 26 15:59:53.918808 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:53.918508    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cde8a1d7-d445-4d0b-ba25-dc3e56d6657b-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kvcvx\" (UID: \"cde8a1d7-d445-4d0b-ba25-dc3e56d6657b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kvcvx"
Jul 26 15:59:53.918808 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:53.918553    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cde8a1d7-d445-4d0b-ba25-dc3e56d6657b-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kvcvx\" (UID: \"cde8a1d7-d445-4d0b-ba25-dc3e56d6657b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kvcvx"
Jul 26 15:59:53.918808 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:53.918596    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cde8a1d7-d445-4d0b-ba25-dc3e56d6657b-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kvcvx\" (UID: \"cde8a1d7-d445-4d0b-ba25-dc3e56d6657b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kvcvx"
Jul 26 15:59:54.019909 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:54.019876    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zds82\" (UniqueName: \"kubernetes.io/projected/cde8a1d7-d445-4d0b-ba25-dc3e56d6657b-kube-api-access-zds82\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kvcvx\" (UID: \"cde8a1d7-d445-4d0b-ba25-dc3e56d6657b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kvcvx"
Jul 26 15:59:54.019909 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:54.019914    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cde8a1d7-d445-4d0b-ba25-dc3e56d6657b-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kvcvx\" (UID: \"cde8a1d7-d445-4d0b-ba25-dc3e56d6657b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kvcvx"
Jul 26 15:59:54.020178 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:54.019954    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cde8a1d7-d445-4d0b-ba25-dc3e56d6657b-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kvcvx\" (UID: \"cde8a1d7-d445-4d0b-ba25-dc3e56d6657b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kvcvx"
Jul 26 15:59:54.020178 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:54.020030    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cde8a1d7-d445-4d0b-ba25-dc3e56d6657b-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kvcvx\" (UID: \"cde8a1d7-d445-4d0b-ba25-dc3e56d6657b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kvcvx"
Jul 26 15:59:54.020178 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:54.020093    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cde8a1d7-d445-4d0b-ba25-dc3e56d6657b-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kvcvx\" (UID: \"cde8a1d7-d445-4d0b-ba25-dc3e56d6657b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kvcvx"
Jul 26 15:59:54.020178 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:54.020169    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cde8a1d7-d445-4d0b-ba25-dc3e56d6657b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kvcvx\" (UID: \"cde8a1d7-d445-4d0b-ba25-dc3e56d6657b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kvcvx"
Jul 26 15:59:54.020736 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:54.020377    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cde8a1d7-d445-4d0b-ba25-dc3e56d6657b-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kvcvx\" (UID: \"cde8a1d7-d445-4d0b-ba25-dc3e56d6657b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kvcvx"
Jul 26 15:59:54.020736 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:54.020499    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cde8a1d7-d445-4d0b-ba25-dc3e56d6657b-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kvcvx\" (UID: \"cde8a1d7-d445-4d0b-ba25-dc3e56d6657b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kvcvx"
Jul 26 15:59:54.020736 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:54.020551    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cde8a1d7-d445-4d0b-ba25-dc3e56d6657b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kvcvx\" (UID: \"cde8a1d7-d445-4d0b-ba25-dc3e56d6657b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kvcvx"
Jul 26 15:59:54.022381 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:54.022363    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cde8a1d7-d445-4d0b-ba25-dc3e56d6657b-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kvcvx\" (UID: \"cde8a1d7-d445-4d0b-ba25-dc3e56d6657b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kvcvx"
Jul 26 15:59:54.022564 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:54.022546    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cde8a1d7-d445-4d0b-ba25-dc3e56d6657b-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kvcvx\" (UID: \"cde8a1d7-d445-4d0b-ba25-dc3e56d6657b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kvcvx"
Jul 26 15:59:54.028022 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:54.027999    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zds82\" (UniqueName: \"kubernetes.io/projected/cde8a1d7-d445-4d0b-ba25-dc3e56d6657b-kube-api-access-zds82\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kvcvx\" (UID: \"cde8a1d7-d445-4d0b-ba25-dc3e56d6657b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kvcvx"
Jul 26 15:59:54.166611 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:54.166524    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kvcvx"
Jul 26 15:59:54.307079 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:54.307054    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-kvcvx"]
Jul 26 15:59:54.308732 ip-10-0-133-204 kubenswrapper[2563]: W0726 15:59:54.308701    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcde8a1d7_d445_4d0b_ba25_dc3e56d6657b.slice/crio-2bccf26f847f9fe87b0baf472e62d0059b30128e5b58bc3e977f4553d9651975 WatchSource:0}: Error finding container 2bccf26f847f9fe87b0baf472e62d0059b30128e5b58bc3e977f4553d9651975: Status 404 returned error can't find the container with id 2bccf26f847f9fe87b0baf472e62d0059b30128e5b58bc3e977f4553d9651975
Jul 26 15:59:55.149594 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:55.149560    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kvcvx" event={"ID":"cde8a1d7-d445-4d0b-ba25-dc3e56d6657b","Type":"ContainerStarted","Data":"a9b2e2d0cb74410c9d13e7c4cca2f6235133aa6c5d9704aea7e14c6f7707472b"}
Jul 26 15:59:55.150003 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:55.149610    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kvcvx" event={"ID":"cde8a1d7-d445-4d0b-ba25-dc3e56d6657b","Type":"ContainerStarted","Data":"2bccf26f847f9fe87b0baf472e62d0059b30128e5b58bc3e977f4553d9651975"}
Jul 26 15:59:55.169867 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:55.169809    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kvcvx" podStartSLOduration=2.00618593 podStartE2EDuration="2.169787906s" totalImagesPullingTime="163.601976ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:59:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:59:55.168597392 +0000 UTC m=+587.272823210" watchObservedRunningTime="2026-07-26 15:59:55.169787906 +0000 UTC m=+587.274013714"
Jul 26 15:59:55.871731 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:55.871693    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mlpjn"
Jul 26 15:59:55.871946 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:55.871749    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mlpjn"
Jul 26 15:59:55.884558 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:55.884532    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mlpjn"
Jul 26 15:59:56.165667 ip-10-0-133-204 kubenswrapper[2563]: I0726 15:59:56.165579    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mlpjn"
Jul 26 16:00:04.167616 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:00:04.167581    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kvcvx"
Jul 26 16:00:04.168059 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:00:04.167667    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kvcvx"
Jul 26 16:00:04.180041 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:00:04.180014    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kvcvx"
Jul 26 16:00:04.208801 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:00:04.208755    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kvcvx"
Jul 26 16:00:08.080900 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:00:08.080864    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:00:24.490025 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:00:24.489991    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:00:37.487756 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:00:37.487638    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:00:40.739697 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:00:40.739647    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:00:42.281699 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:00:42.281663    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:00:42.381508 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:00:42.381464    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:02:15.233973 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:02:15.233942    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:02:15.320535 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:02:15.320488    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:02:16.301210 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:02:16.301180    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:02:16.416790 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:02:16.416731    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:02:20.572888 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:02:20.572853    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:02:21.734477 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:02:21.734443    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:02:21.835756 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:02:21.835717    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:03:03.621752 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:03:03.621714    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:03:04.456355 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:03:04.456326    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5fb95c8f7b-2xbnv"]
Jul 26 16:03:04.456598 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:03:04.456571    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5fb95c8f7b-2xbnv" podUID="2f11f601-c407-4354-874b-8a83a50b725d" containerName="manager" containerID="cri-o://780d503446aa2ae674c2eb66b9c391c34ea7c42d0c922f625217fb9e4ace0407" gracePeriod=30
Jul 26 16:03:04.705254 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:03:04.705232    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fb95c8f7b-2xbnv"
Jul 26 16:03:04.786939 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:03:04.786909    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8hqr7\" (UniqueName: \"kubernetes.io/projected/2f11f601-c407-4354-874b-8a83a50b725d-kube-api-access-8hqr7\") pod \"2f11f601-c407-4354-874b-8a83a50b725d\" (UID: \"2f11f601-c407-4354-874b-8a83a50b725d\") "
Jul 26 16:03:04.787139 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:03:04.786972    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2f11f601-c407-4354-874b-8a83a50b725d-webhook-cert\") pod \"2f11f601-c407-4354-874b-8a83a50b725d\" (UID: \"2f11f601-c407-4354-874b-8a83a50b725d\") "
Jul 26 16:03:04.788968 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:03:04.788934    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f11f601-c407-4354-874b-8a83a50b725d-kube-api-access-8hqr7" (OuterVolumeSpecName: "kube-api-access-8hqr7") pod "2f11f601-c407-4354-874b-8a83a50b725d" (UID: "2f11f601-c407-4354-874b-8a83a50b725d"). InnerVolumeSpecName "kube-api-access-8hqr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 26 16:03:04.789070 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:03:04.789010    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f11f601-c407-4354-874b-8a83a50b725d-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "2f11f601-c407-4354-874b-8a83a50b725d" (UID: "2f11f601-c407-4354-874b-8a83a50b725d"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 26 16:03:04.887911 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:03:04.887872    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hqr7\" (UniqueName: \"kubernetes.io/projected/2f11f601-c407-4354-874b-8a83a50b725d-kube-api-access-8hqr7\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:03:04.887911 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:03:04.887905    2563 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2f11f601-c407-4354-874b-8a83a50b725d-webhook-cert\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:03:05.088458 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:03:05.088357    2563 generic.go:356] "Generic (PLEG): container finished" podID="2f11f601-c407-4354-874b-8a83a50b725d" containerID="780d503446aa2ae674c2eb66b9c391c34ea7c42d0c922f625217fb9e4ace0407" exitCode=0
Jul 26 16:03:05.088458 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:03:05.088422    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fb95c8f7b-2xbnv"
Jul 26 16:03:05.088696 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:03:05.088410    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fb95c8f7b-2xbnv" event={"ID":"2f11f601-c407-4354-874b-8a83a50b725d","Type":"ContainerDied","Data":"780d503446aa2ae674c2eb66b9c391c34ea7c42d0c922f625217fb9e4ace0407"}
Jul 26 16:03:05.088696 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:03:05.088561    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fb95c8f7b-2xbnv" event={"ID":"2f11f601-c407-4354-874b-8a83a50b725d","Type":"ContainerDied","Data":"65c5179fcd3f44cb41935c825e51773aaaf90a246ede0907cc8fa76e53fc45fe"}
Jul 26 16:03:05.088696 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:03:05.088595    2563 scope.go:117] "RemoveContainer" containerID="780d503446aa2ae674c2eb66b9c391c34ea7c42d0c922f625217fb9e4ace0407"
Jul 26 16:03:05.098123 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:03:05.098106    2563 scope.go:117] "RemoveContainer" containerID="780d503446aa2ae674c2eb66b9c391c34ea7c42d0c922f625217fb9e4ace0407"
Jul 26 16:03:05.098401 ip-10-0-133-204 kubenswrapper[2563]: E0726 16:03:05.098382    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"780d503446aa2ae674c2eb66b9c391c34ea7c42d0c922f625217fb9e4ace0407\": container with ID starting with 780d503446aa2ae674c2eb66b9c391c34ea7c42d0c922f625217fb9e4ace0407 not found: ID does not exist" containerID="780d503446aa2ae674c2eb66b9c391c34ea7c42d0c922f625217fb9e4ace0407"
Jul 26 16:03:05.098445 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:03:05.098410    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"780d503446aa2ae674c2eb66b9c391c34ea7c42d0c922f625217fb9e4ace0407"} err="failed to get container status \"780d503446aa2ae674c2eb66b9c391c34ea7c42d0c922f625217fb9e4ace0407\": rpc error: code = NotFound desc = could not find container \"780d503446aa2ae674c2eb66b9c391c34ea7c42d0c922f625217fb9e4ace0407\": container with ID starting with 780d503446aa2ae674c2eb66b9c391c34ea7c42d0c922f625217fb9e4ace0407 not found: ID does not exist"
Jul 26 16:03:05.109959 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:03:05.109930    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5fb95c8f7b-2xbnv"]
Jul 26 16:03:05.114087 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:03:05.114064    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5fb95c8f7b-2xbnv"]
Jul 26 16:03:06.378920 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:03:06.378890    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f11f601-c407-4354-874b-8a83a50b725d" path="/var/lib/kubelet/pods/2f11f601-c407-4354-874b-8a83a50b725d/volumes"
Jul 26 16:04:05.905263 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:04:05.905228    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5fb95c8f7b-9kdzf"]
Jul 26 16:04:05.905840 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:04:05.905826    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f11f601-c407-4354-874b-8a83a50b725d" containerName="manager"
Jul 26 16:04:05.905840 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:04:05.905841    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f11f601-c407-4354-874b-8a83a50b725d" containerName="manager"
Jul 26 16:04:05.905925 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:04:05.905892    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f11f601-c407-4354-874b-8a83a50b725d" containerName="manager"
Jul 26 16:04:05.908802 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:04:05.908785    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fb95c8f7b-9kdzf"
Jul 26 16:04:05.911881 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:04:05.911861    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 26 16:04:05.912000 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:04:05.911916    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 26 16:04:05.912000 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:04:05.911924    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-dc6f7\""
Jul 26 16:04:05.916952 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:04:05.916547    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5fb95c8f7b-9kdzf"]
Jul 26 16:04:06.052744 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:04:06.052699    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/271385d7-7d3b-42df-a3d5-0a02058df803-webhook-cert\") pod \"maas-controller-5fb95c8f7b-9kdzf\" (UID: \"271385d7-7d3b-42df-a3d5-0a02058df803\") " pod="opendatahub/maas-controller-5fb95c8f7b-9kdzf"
Jul 26 16:04:06.052942 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:04:06.052795    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkstb\" (UniqueName: \"kubernetes.io/projected/271385d7-7d3b-42df-a3d5-0a02058df803-kube-api-access-bkstb\") pod \"maas-controller-5fb95c8f7b-9kdzf\" (UID: \"271385d7-7d3b-42df-a3d5-0a02058df803\") " pod="opendatahub/maas-controller-5fb95c8f7b-9kdzf"
Jul 26 16:04:06.154137 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:04:06.154103    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkstb\" (UniqueName: \"kubernetes.io/projected/271385d7-7d3b-42df-a3d5-0a02058df803-kube-api-access-bkstb\") pod \"maas-controller-5fb95c8f7b-9kdzf\" (UID: \"271385d7-7d3b-42df-a3d5-0a02058df803\") " pod="opendatahub/maas-controller-5fb95c8f7b-9kdzf"
Jul 26 16:04:06.154309 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:04:06.154191    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/271385d7-7d3b-42df-a3d5-0a02058df803-webhook-cert\") pod \"maas-controller-5fb95c8f7b-9kdzf\" (UID: \"271385d7-7d3b-42df-a3d5-0a02058df803\") " pod="opendatahub/maas-controller-5fb95c8f7b-9kdzf"
Jul 26 16:04:06.156652 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:04:06.156586    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/271385d7-7d3b-42df-a3d5-0a02058df803-webhook-cert\") pod \"maas-controller-5fb95c8f7b-9kdzf\" (UID: \"271385d7-7d3b-42df-a3d5-0a02058df803\") " pod="opendatahub/maas-controller-5fb95c8f7b-9kdzf"
Jul 26 16:04:06.161553 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:04:06.161529    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkstb\" (UniqueName: \"kubernetes.io/projected/271385d7-7d3b-42df-a3d5-0a02058df803-kube-api-access-bkstb\") pod \"maas-controller-5fb95c8f7b-9kdzf\" (UID: \"271385d7-7d3b-42df-a3d5-0a02058df803\") " pod="opendatahub/maas-controller-5fb95c8f7b-9kdzf"
Jul 26 16:04:06.221610 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:04:06.221573    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fb95c8f7b-9kdzf"
Jul 26 16:04:06.350278 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:04:06.350244    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5fb95c8f7b-9kdzf"]
Jul 26 16:04:06.353627 ip-10-0-133-204 kubenswrapper[2563]: W0726 16:04:06.353600    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod271385d7_7d3b_42df_a3d5_0a02058df803.slice/crio-e00b53672ee1aa8051a938682f8dce17ec0a5f6fd1af9d9e44f1d76399f2940f WatchSource:0}: Error finding container e00b53672ee1aa8051a938682f8dce17ec0a5f6fd1af9d9e44f1d76399f2940f: Status 404 returned error can't find the container with id e00b53672ee1aa8051a938682f8dce17ec0a5f6fd1af9d9e44f1d76399f2940f
Jul 26 16:04:06.354933 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:04:06.354916    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 26 16:04:06.379630 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:04:06.379603    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fb95c8f7b-9kdzf" event={"ID":"271385d7-7d3b-42df-a3d5-0a02058df803","Type":"ContainerStarted","Data":"e00b53672ee1aa8051a938682f8dce17ec0a5f6fd1af9d9e44f1d76399f2940f"}
Jul 26 16:04:07.386152 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:04:07.386071    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fb95c8f7b-9kdzf" event={"ID":"271385d7-7d3b-42df-a3d5-0a02058df803","Type":"ContainerStarted","Data":"f1472773a5d75f238104687639fc2f012570e0eec25e2e8a2a35c7603ff2f4ad"}
Jul 26 16:04:07.386500 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:04:07.386173    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5fb95c8f7b-9kdzf"
Jul 26 16:04:07.403471 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:04:07.403429    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5fb95c8f7b-9kdzf" podStartSLOduration=1.784732042 podStartE2EDuration="2.403416074s" totalImagesPullingTime="618.684032ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:04:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:04:07.402442575 +0000 UTC m=+839.506668383" watchObservedRunningTime="2026-07-26 16:04:07.403416074 +0000 UTC m=+839.507641881"
Jul 26 16:04:09.741058 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:04:09.741015    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:04:18.393618 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:04:18.393581    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5fb95c8f7b-9kdzf"
Jul 26 16:04:22.910575 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:04:22.910535    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:04:23.007807 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:04:23.007771    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5fb95c8f7b-9kdzf"]
Jul 26 16:04:23.008006 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:04:23.007983    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5fb95c8f7b-9kdzf" podUID="271385d7-7d3b-42df-a3d5-0a02058df803" containerName="manager" containerID="cri-o://f1472773a5d75f238104687639fc2f012570e0eec25e2e8a2a35c7603ff2f4ad" gracePeriod=30
Jul 26 16:04:23.258586 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:04:23.258561    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fb95c8f7b-9kdzf"
Jul 26 16:04:23.307110 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:04:23.307080    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/271385d7-7d3b-42df-a3d5-0a02058df803-webhook-cert\") pod \"271385d7-7d3b-42df-a3d5-0a02058df803\" (UID: \"271385d7-7d3b-42df-a3d5-0a02058df803\") "
Jul 26 16:04:23.307285 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:04:23.307141    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bkstb\" (UniqueName: \"kubernetes.io/projected/271385d7-7d3b-42df-a3d5-0a02058df803-kube-api-access-bkstb\") pod \"271385d7-7d3b-42df-a3d5-0a02058df803\" (UID: \"271385d7-7d3b-42df-a3d5-0a02058df803\") "
Jul 26 16:04:23.309199 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:04:23.309161    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/271385d7-7d3b-42df-a3d5-0a02058df803-kube-api-access-bkstb" (OuterVolumeSpecName: "kube-api-access-bkstb") pod "271385d7-7d3b-42df-a3d5-0a02058df803" (UID: "271385d7-7d3b-42df-a3d5-0a02058df803"). InnerVolumeSpecName "kube-api-access-bkstb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 26 16:04:23.309324 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:04:23.309254    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/271385d7-7d3b-42df-a3d5-0a02058df803-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "271385d7-7d3b-42df-a3d5-0a02058df803" (UID: "271385d7-7d3b-42df-a3d5-0a02058df803"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 26 16:04:23.408898 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:04:23.408862    2563 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/271385d7-7d3b-42df-a3d5-0a02058df803-webhook-cert\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:04:23.408898 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:04:23.408891    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkstb\" (UniqueName: \"kubernetes.io/projected/271385d7-7d3b-42df-a3d5-0a02058df803-kube-api-access-bkstb\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:04:23.459976 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:04:23.459882    2563 generic.go:356] "Generic (PLEG): container finished" podID="271385d7-7d3b-42df-a3d5-0a02058df803" containerID="f1472773a5d75f238104687639fc2f012570e0eec25e2e8a2a35c7603ff2f4ad" exitCode=0
Jul 26 16:04:23.459976 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:04:23.459925    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fb95c8f7b-9kdzf"
Jul 26 16:04:23.459976 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:04:23.459969    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fb95c8f7b-9kdzf" event={"ID":"271385d7-7d3b-42df-a3d5-0a02058df803","Type":"ContainerDied","Data":"f1472773a5d75f238104687639fc2f012570e0eec25e2e8a2a35c7603ff2f4ad"}
Jul 26 16:04:23.460189 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:04:23.460018    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fb95c8f7b-9kdzf" event={"ID":"271385d7-7d3b-42df-a3d5-0a02058df803","Type":"ContainerDied","Data":"e00b53672ee1aa8051a938682f8dce17ec0a5f6fd1af9d9e44f1d76399f2940f"}
Jul 26 16:04:23.460189 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:04:23.460041    2563 scope.go:117] "RemoveContainer" containerID="f1472773a5d75f238104687639fc2f012570e0eec25e2e8a2a35c7603ff2f4ad"
Jul 26 16:04:23.469349 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:04:23.469334    2563 scope.go:117] "RemoveContainer" containerID="f1472773a5d75f238104687639fc2f012570e0eec25e2e8a2a35c7603ff2f4ad"
Jul 26 16:04:23.469607 ip-10-0-133-204 kubenswrapper[2563]: E0726 16:04:23.469589    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1472773a5d75f238104687639fc2f012570e0eec25e2e8a2a35c7603ff2f4ad\": container with ID starting with f1472773a5d75f238104687639fc2f012570e0eec25e2e8a2a35c7603ff2f4ad not found: ID does not exist" containerID="f1472773a5d75f238104687639fc2f012570e0eec25e2e8a2a35c7603ff2f4ad"
Jul 26 16:04:23.469654 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:04:23.469617    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1472773a5d75f238104687639fc2f012570e0eec25e2e8a2a35c7603ff2f4ad"} err="failed to get container status \"f1472773a5d75f238104687639fc2f012570e0eec25e2e8a2a35c7603ff2f4ad\": rpc error: code = NotFound desc = could not find container \"f1472773a5d75f238104687639fc2f012570e0eec25e2e8a2a35c7603ff2f4ad\": container with ID starting with f1472773a5d75f238104687639fc2f012570e0eec25e2e8a2a35c7603ff2f4ad not found: ID does not exist"
Jul 26 16:04:23.481402 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:04:23.481372    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5fb95c8f7b-9kdzf"]
Jul 26 16:04:23.483009 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:04:23.482991    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5fb95c8f7b-9kdzf"]
Jul 26 16:04:24.141323 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:04:24.141291    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:04:24.247022 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:04:24.246988    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5fb95c8f7b-gbwfs"]
Jul 26 16:04:24.247532 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:04:24.247517    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="271385d7-7d3b-42df-a3d5-0a02058df803" containerName="manager"
Jul 26 16:04:24.247577 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:04:24.247535    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="271385d7-7d3b-42df-a3d5-0a02058df803" containerName="manager"
Jul 26 16:04:24.247636 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:04:24.247625    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="271385d7-7d3b-42df-a3d5-0a02058df803" containerName="manager"
Jul 26 16:04:24.252356 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:04:24.252332    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fb95c8f7b-gbwfs"
Jul 26 16:04:24.255775 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:04:24.255425    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-dc6f7\""
Jul 26 16:04:24.255775 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:04:24.255474    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 26 16:04:24.255964 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:04:24.255891    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 26 16:04:24.258878 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:04:24.258854    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5fb95c8f7b-gbwfs"]
Jul 26 16:04:24.317162 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:04:24.317125    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c625fd17-2520-4288-8f6c-efb9f6b9d79c-webhook-cert\") pod \"maas-controller-5fb95c8f7b-gbwfs\" (UID: \"c625fd17-2520-4288-8f6c-efb9f6b9d79c\") " pod="opendatahub/maas-controller-5fb95c8f7b-gbwfs"
Jul 26 16:04:24.317162 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:04:24.317162    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdn4m\" (UniqueName: \"kubernetes.io/projected/c625fd17-2520-4288-8f6c-efb9f6b9d79c-kube-api-access-gdn4m\") pod \"maas-controller-5fb95c8f7b-gbwfs\" (UID: \"c625fd17-2520-4288-8f6c-efb9f6b9d79c\") " pod="opendatahub/maas-controller-5fb95c8f7b-gbwfs"
Jul 26 16:04:24.373612 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:04:24.373578    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="271385d7-7d3b-42df-a3d5-0a02058df803" path="/var/lib/kubelet/pods/271385d7-7d3b-42df-a3d5-0a02058df803/volumes"
Jul 26 16:04:24.417629 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:04:24.417555    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c625fd17-2520-4288-8f6c-efb9f6b9d79c-webhook-cert\") pod \"maas-controller-5fb95c8f7b-gbwfs\" (UID: \"c625fd17-2520-4288-8f6c-efb9f6b9d79c\") " pod="opendatahub/maas-controller-5fb95c8f7b-gbwfs"
Jul 26 16:04:24.417629 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:04:24.417590    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdn4m\" (UniqueName: \"kubernetes.io/projected/c625fd17-2520-4288-8f6c-efb9f6b9d79c-kube-api-access-gdn4m\") pod \"maas-controller-5fb95c8f7b-gbwfs\" (UID: \"c625fd17-2520-4288-8f6c-efb9f6b9d79c\") " pod="opendatahub/maas-controller-5fb95c8f7b-gbwfs"
Jul 26 16:04:24.420040 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:04:24.420012    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c625fd17-2520-4288-8f6c-efb9f6b9d79c-webhook-cert\") pod \"maas-controller-5fb95c8f7b-gbwfs\" (UID: \"c625fd17-2520-4288-8f6c-efb9f6b9d79c\") " pod="opendatahub/maas-controller-5fb95c8f7b-gbwfs"
Jul 26 16:04:24.425306 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:04:24.425279    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdn4m\" (UniqueName: \"kubernetes.io/projected/c625fd17-2520-4288-8f6c-efb9f6b9d79c-kube-api-access-gdn4m\") pod \"maas-controller-5fb95c8f7b-gbwfs\" (UID: \"c625fd17-2520-4288-8f6c-efb9f6b9d79c\") " pod="opendatahub/maas-controller-5fb95c8f7b-gbwfs"
Jul 26 16:04:24.566397 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:04:24.566359    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fb95c8f7b-gbwfs"
Jul 26 16:04:24.697076 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:04:24.697050    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5fb95c8f7b-gbwfs"]
Jul 26 16:04:24.699859 ip-10-0-133-204 kubenswrapper[2563]: W0726 16:04:24.699821    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc625fd17_2520_4288_8f6c_efb9f6b9d79c.slice/crio-88c84ec41be22b7cd82660294535e5efe46560fa799519f2d5593110ba4ef88f WatchSource:0}: Error finding container 88c84ec41be22b7cd82660294535e5efe46560fa799519f2d5593110ba4ef88f: Status 404 returned error can't find the container with id 88c84ec41be22b7cd82660294535e5efe46560fa799519f2d5593110ba4ef88f
Jul 26 16:04:25.472483 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:04:25.472399    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fb95c8f7b-gbwfs" event={"ID":"c625fd17-2520-4288-8f6c-efb9f6b9d79c","Type":"ContainerStarted","Data":"339231a81dc88045fd05b14df184fde573c2fc72bcd76f0191e4962b22f1fcdb"}
Jul 26 16:04:25.472483 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:04:25.472448    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fb95c8f7b-gbwfs" event={"ID":"c625fd17-2520-4288-8f6c-efb9f6b9d79c","Type":"ContainerStarted","Data":"88c84ec41be22b7cd82660294535e5efe46560fa799519f2d5593110ba4ef88f"}
Jul 26 16:04:25.472921 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:04:25.472519    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5fb95c8f7b-gbwfs"
Jul 26 16:04:25.490161 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:04:25.490084    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5fb95c8f7b-gbwfs" podStartSLOduration=1.060154434 podStartE2EDuration="1.490062055s" totalImagesPullingTime="429.907621ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:04:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:04:25.487493463 +0000 UTC m=+857.591719274" watchObservedRunningTime="2026-07-26 16:04:25.490062055 +0000 UTC m=+857.594287864"
Jul 26 16:04:36.482284 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:04:36.482252    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5fb95c8f7b-gbwfs"
Jul 26 16:04:54.532829 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:04:54.532791    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:04:54.631261 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:04:54.631221    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:04:54.731582 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:04:54.731549    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:05:00.383871 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:05:00.383789    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:05:00.447066 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:05:00.447030    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:05:00.526089 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:05:00.526054    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:05:00.625998 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:05:00.625968    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:05:00.725754 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:05:00.725727    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:05:06.419195 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:05:06.419160    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:05:06.519881 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:05:06.519845    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:05:44.304714 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:05:44.304674    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:07:38.746363 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:07:38.746327    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:07:38.833000 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:07:38.832962    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:09:04.030692 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:09:04.030647    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2wt74"]
Jul 26 16:09:04.031323 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:09:04.031011    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2wt74" podUID="6ac3f9b7-ebaf-41ad-ba2d-70c2ef6e81d3" containerName="manager" containerID="cri-o://64365ac7382326182fec1d258749b919ae7357cf895a0ee6e97d863ecd01958b" gracePeriod=10
Jul 26 16:09:07.819693 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:09:07.819662    2563 generic.go:356] "Generic (PLEG): container finished" podID="6ac3f9b7-ebaf-41ad-ba2d-70c2ef6e81d3" containerID="64365ac7382326182fec1d258749b919ae7357cf895a0ee6e97d863ecd01958b" exitCode=0
Jul 26 16:09:07.820073 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:09:07.819724    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2wt74" event={"ID":"6ac3f9b7-ebaf-41ad-ba2d-70c2ef6e81d3","Type":"ContainerDied","Data":"64365ac7382326182fec1d258749b919ae7357cf895a0ee6e97d863ecd01958b"}
Jul 26 16:09:07.896597 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:09:07.896574    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2wt74"
Jul 26 16:09:08.038472 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:09:08.038390    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6ac3f9b7-ebaf-41ad-ba2d-70c2ef6e81d3-extensions-socket-volume\") pod \"6ac3f9b7-ebaf-41ad-ba2d-70c2ef6e81d3\" (UID: \"6ac3f9b7-ebaf-41ad-ba2d-70c2ef6e81d3\") "
Jul 26 16:09:08.038472 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:09:08.038428    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-664qz\" (UniqueName: \"kubernetes.io/projected/6ac3f9b7-ebaf-41ad-ba2d-70c2ef6e81d3-kube-api-access-664qz\") pod \"6ac3f9b7-ebaf-41ad-ba2d-70c2ef6e81d3\" (UID: \"6ac3f9b7-ebaf-41ad-ba2d-70c2ef6e81d3\") "
Jul 26 16:09:08.038815 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:09:08.038757    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ac3f9b7-ebaf-41ad-ba2d-70c2ef6e81d3-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "6ac3f9b7-ebaf-41ad-ba2d-70c2ef6e81d3" (UID: "6ac3f9b7-ebaf-41ad-ba2d-70c2ef6e81d3"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 26 16:09:08.040452 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:09:08.040434    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ac3f9b7-ebaf-41ad-ba2d-70c2ef6e81d3-kube-api-access-664qz" (OuterVolumeSpecName: "kube-api-access-664qz") pod "6ac3f9b7-ebaf-41ad-ba2d-70c2ef6e81d3" (UID: "6ac3f9b7-ebaf-41ad-ba2d-70c2ef6e81d3"). InnerVolumeSpecName "kube-api-access-664qz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 26 16:09:08.140068 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:09:08.140025    2563 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6ac3f9b7-ebaf-41ad-ba2d-70c2ef6e81d3-extensions-socket-volume\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:09:08.140068 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:09:08.140063    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-664qz\" (UniqueName: \"kubernetes.io/projected/6ac3f9b7-ebaf-41ad-ba2d-70c2ef6e81d3-kube-api-access-664qz\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:09:08.390113 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:09:08.390086    2563 scope.go:117] "RemoveContainer" containerID="64365ac7382326182fec1d258749b919ae7357cf895a0ee6e97d863ecd01958b"
Jul 26 16:09:08.824594 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:09:08.824559    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2wt74"
Jul 26 16:09:08.825002 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:09:08.824566    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2wt74" event={"ID":"6ac3f9b7-ebaf-41ad-ba2d-70c2ef6e81d3","Type":"ContainerDied","Data":"d603213b2e4ea782cb8238fe71ff8e1db35be47ba2f9152437a767cf796add85"}
Jul 26 16:09:08.855685 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:09:08.855643    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2wt74"]
Jul 26 16:09:08.860992 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:09:08.860959    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2wt74"]
Jul 26 16:09:10.372622 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:09:10.372583    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ac3f9b7-ebaf-41ad-ba2d-70c2ef6e81d3" path="/var/lib/kubelet/pods/6ac3f9b7-ebaf-41ad-ba2d-70c2ef6e81d3/volumes"
Jul 26 16:09:57.996205 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:09:57.996120    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:10:05.207578 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:10:05.207533    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-54d5v"]
Jul 26 16:10:05.208016 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:10:05.207958    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ac3f9b7-ebaf-41ad-ba2d-70c2ef6e81d3" containerName="manager"
Jul 26 16:10:05.208016 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:10:05.207970    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ac3f9b7-ebaf-41ad-ba2d-70c2ef6e81d3" containerName="manager"
Jul 26 16:10:05.208104 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:10:05.208026    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ac3f9b7-ebaf-41ad-ba2d-70c2ef6e81d3" containerName="manager"
Jul 26 16:10:05.211087 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:10:05.211071    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-54d5v"
Jul 26 16:10:05.213581 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:10:05.213552    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-x8dzq\""
Jul 26 16:10:05.213705 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:10:05.213642    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 26 16:10:05.213705 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:10:05.213640    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 26 16:10:05.225976 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:10:05.225947    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-54d5v"]
Jul 26 16:10:05.365112 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:10:05.365076    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4e276424-a20b-4eaa-b908-6d17df7bfcb3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-54d5v\" (UID: \"4e276424-a20b-4eaa-b908-6d17df7bfcb3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-54d5v"
Jul 26 16:10:05.365296 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:10:05.365139    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wfvr\" (UniqueName: \"kubernetes.io/projected/4e276424-a20b-4eaa-b908-6d17df7bfcb3-kube-api-access-4wfvr\") pod \"kuadrant-operator-controller-manager-86878c769f-54d5v\" (UID: \"4e276424-a20b-4eaa-b908-6d17df7bfcb3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-54d5v"
Jul 26 16:10:05.466342 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:10:05.466237    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4e276424-a20b-4eaa-b908-6d17df7bfcb3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-54d5v\" (UID: \"4e276424-a20b-4eaa-b908-6d17df7bfcb3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-54d5v"
Jul 26 16:10:05.466342 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:10:05.466303    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wfvr\" (UniqueName: \"kubernetes.io/projected/4e276424-a20b-4eaa-b908-6d17df7bfcb3-kube-api-access-4wfvr\") pod \"kuadrant-operator-controller-manager-86878c769f-54d5v\" (UID: \"4e276424-a20b-4eaa-b908-6d17df7bfcb3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-54d5v"
Jul 26 16:10:05.466733 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:10:05.466708    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4e276424-a20b-4eaa-b908-6d17df7bfcb3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-54d5v\" (UID: \"4e276424-a20b-4eaa-b908-6d17df7bfcb3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-54d5v"
Jul 26 16:10:05.475157 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:10:05.475132    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wfvr\" (UniqueName: \"kubernetes.io/projected/4e276424-a20b-4eaa-b908-6d17df7bfcb3-kube-api-access-4wfvr\") pod \"kuadrant-operator-controller-manager-86878c769f-54d5v\" (UID: \"4e276424-a20b-4eaa-b908-6d17df7bfcb3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-54d5v"
Jul 26 16:10:05.521903 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:10:05.521861    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-54d5v"
Jul 26 16:10:05.674338 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:10:05.674314    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-54d5v"]
Jul 26 16:10:05.676266 ip-10-0-133-204 kubenswrapper[2563]: W0726 16:10:05.676235    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e276424_a20b_4eaa_b908_6d17df7bfcb3.slice/crio-286635f603cb9fe760481a670845d91a43240ae4c837bd42d077e36c131a66e7 WatchSource:0}: Error finding container 286635f603cb9fe760481a670845d91a43240ae4c837bd42d077e36c131a66e7: Status 404 returned error can't find the container with id 286635f603cb9fe760481a670845d91a43240ae4c837bd42d077e36c131a66e7
Jul 26 16:10:06.095673 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:10:06.095631    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-54d5v" event={"ID":"4e276424-a20b-4eaa-b908-6d17df7bfcb3","Type":"ContainerStarted","Data":"ef4bb130add2a6c9ebb2d92b8f845ddde5417a21d98dd679d225f6426a6a8340"}
Jul 26 16:10:06.095899 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:10:06.095750    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-54d5v"
Jul 26 16:10:06.095899 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:10:06.095783    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-54d5v" event={"ID":"4e276424-a20b-4eaa-b908-6d17df7bfcb3","Type":"ContainerStarted","Data":"286635f603cb9fe760481a670845d91a43240ae4c837bd42d077e36c131a66e7"}
Jul 26 16:10:06.113533 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:10:06.113470    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-54d5v" podStartSLOduration=1.113450909 podStartE2EDuration="1.113450909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:10:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:10:06.11150233 +0000 UTC m=+1198.215728139" watchObservedRunningTime="2026-07-26 16:10:06.113450909 +0000 UTC m=+1198.217676720"
Jul 26 16:10:17.103448 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:10:17.103417    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-54d5v"
Jul 26 16:12:13.008798 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:12:13.008744    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:12:14.480793 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:12:14.480740    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:12:15.187206 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:12:15.187172    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:13:02.254364 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:13:02.254268    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:13:08.763127 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:13:08.763085    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:13:36.985105 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:13:36.985066    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:13:38.177254 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:13:38.177200    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:16:51.310515 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:16:51.310484    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:17:09.946731 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:09.946697    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:17:13.155324 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:13.155194    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:17:13.261388 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:13.261357    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:17:13.694351 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:13.694307    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:17:19.341771 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:19.341716    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:17:20.547143 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:20.547109    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:17:22.309902 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:22.309854    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:17:22.397643 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:22.397611    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:17:22.558663 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:22.558629    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6abd1847-58b7c9b654-r849s"]
Jul 26 16:17:22.562360 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:22.562292    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6abd1847-58b7c9b654-r849s"
Jul 26 16:17:22.564569 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:22.564546    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-6abd1847-serving-cert\""
Jul 26 16:17:22.571353 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:22.571328    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6abd1847-58b7c9b654-r849s"]
Jul 26 16:17:22.590408 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:22.590370    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:17:22.729908 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:22.729867    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnsqh\" (UniqueName: \"kubernetes.io/projected/79de5b67-3416-40b1-baf6-2ea1f5599631-kube-api-access-tnsqh\") pod \"maas-api-e2e-ait-6abd1847-58b7c9b654-r849s\" (UID: \"79de5b67-3416-40b1-baf6-2ea1f5599631\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6abd1847-58b7c9b654-r849s"
Jul 26 16:17:22.730090 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:22.729945    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79de5b67-3416-40b1-baf6-2ea1f5599631-openshift-service-ca\") pod \"maas-api-e2e-ait-6abd1847-58b7c9b654-r849s\" (UID: \"79de5b67-3416-40b1-baf6-2ea1f5599631\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6abd1847-58b7c9b654-r849s"
Jul 26 16:17:22.730090 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:22.729999    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79de5b67-3416-40b1-baf6-2ea1f5599631-maas-api-tls\") pod \"maas-api-e2e-ait-6abd1847-58b7c9b654-r849s\" (UID: \"79de5b67-3416-40b1-baf6-2ea1f5599631\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6abd1847-58b7c9b654-r849s"
Jul 26 16:17:22.831138 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:22.831039    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79de5b67-3416-40b1-baf6-2ea1f5599631-maas-api-tls\") pod \"maas-api-e2e-ait-6abd1847-58b7c9b654-r849s\" (UID: \"79de5b67-3416-40b1-baf6-2ea1f5599631\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6abd1847-58b7c9b654-r849s"
Jul 26 16:17:22.831138 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:22.831086    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnsqh\" (UniqueName: \"kubernetes.io/projected/79de5b67-3416-40b1-baf6-2ea1f5599631-kube-api-access-tnsqh\") pod \"maas-api-e2e-ait-6abd1847-58b7c9b654-r849s\" (UID: \"79de5b67-3416-40b1-baf6-2ea1f5599631\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6abd1847-58b7c9b654-r849s"
Jul 26 16:17:22.831138 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:22.831142    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79de5b67-3416-40b1-baf6-2ea1f5599631-openshift-service-ca\") pod \"maas-api-e2e-ait-6abd1847-58b7c9b654-r849s\" (UID: \"79de5b67-3416-40b1-baf6-2ea1f5599631\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6abd1847-58b7c9b654-r849s"
Jul 26 16:17:22.831430 ip-10-0-133-204 kubenswrapper[2563]: E0726 16:17:22.831222    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-6abd1847-serving-cert: secret "maas-api-e2e-ait-6abd1847-serving-cert" not found
Jul 26 16:17:22.831430 ip-10-0-133-204 kubenswrapper[2563]: E0726 16:17:22.831314    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79de5b67-3416-40b1-baf6-2ea1f5599631-maas-api-tls podName:79de5b67-3416-40b1-baf6-2ea1f5599631 nodeName:}" failed. No retries permitted until 2026-07-26 16:17:23.331292567 +0000 UTC m=+1635.435518370 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/79de5b67-3416-40b1-baf6-2ea1f5599631-maas-api-tls") pod "maas-api-e2e-ait-6abd1847-58b7c9b654-r849s" (UID: "79de5b67-3416-40b1-baf6-2ea1f5599631") : secret "maas-api-e2e-ait-6abd1847-serving-cert" not found
Jul 26 16:17:22.832020 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:22.831999    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79de5b67-3416-40b1-baf6-2ea1f5599631-openshift-service-ca\") pod \"maas-api-e2e-ait-6abd1847-58b7c9b654-r849s\" (UID: \"79de5b67-3416-40b1-baf6-2ea1f5599631\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6abd1847-58b7c9b654-r849s"
Jul 26 16:17:22.839872 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:22.839845    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnsqh\" (UniqueName: \"kubernetes.io/projected/79de5b67-3416-40b1-baf6-2ea1f5599631-kube-api-access-tnsqh\") pod \"maas-api-e2e-ait-6abd1847-58b7c9b654-r849s\" (UID: \"79de5b67-3416-40b1-baf6-2ea1f5599631\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6abd1847-58b7c9b654-r849s"
Jul 26 16:17:22.922903 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:22.922867    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:17:23.059069 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:23.059038    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:17:23.159675 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:23.159589    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:17:23.171153 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:23.171112    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6abd1847-8c9cc4b8-vx2gx"]
Jul 26 16:17:23.174990 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:23.174967    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6abd1847-8c9cc4b8-vx2gx"
Jul 26 16:17:23.177960 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:23.177939    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-6abd1847\""
Jul 26 16:17:23.180121 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:23.180098    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-6abd1847-dockercfg-6b6t4\""
Jul 26 16:17:23.190783 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:23.190707    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6abd1847-5c7544d97d-jnvql"]
Jul 26 16:17:23.195022 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:23.194983    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6abd1847-8c9cc4b8-vx2gx"]
Jul 26 16:17:23.195164 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:23.195102    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6abd1847-5c7544d97d-jnvql"
Jul 26 16:17:23.210357 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:23.210326    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6abd1847-5c7544d97d-jnvql"]
Jul 26 16:17:23.336701 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:23.336662    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c5b1b12-fd28-4b1b-9440-525eaca94984-plugins-config-volume\") pod \"payload-processing-e2e-ait-6abd1847-5c7544d97d-jnvql\" (UID: \"4c5b1b12-fd28-4b1b-9440-525eaca94984\") " pod="openshift-ingress/payload-processing-e2e-ait-6abd1847-5c7544d97d-jnvql"
Jul 26 16:17:23.337101 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:23.336791    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnlcd\" (UniqueName: \"kubernetes.io/projected/7c61a6df-8f37-477b-967e-bb400bdde08c-kube-api-access-pnlcd\") pod \"payload-pre-processing-e2e-ait-6abd1847-8c9cc4b8-vx2gx\" (UID: \"7c61a6df-8f37-477b-967e-bb400bdde08c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6abd1847-8c9cc4b8-vx2gx"
Jul 26 16:17:23.337101 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:23.336878    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c61a6df-8f37-477b-967e-bb400bdde08c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6abd1847-8c9cc4b8-vx2gx\" (UID: \"7c61a6df-8f37-477b-967e-bb400bdde08c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6abd1847-8c9cc4b8-vx2gx"
Jul 26 16:17:23.337101 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:23.336904    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7492t\" (UniqueName: \"kubernetes.io/projected/4c5b1b12-fd28-4b1b-9440-525eaca94984-kube-api-access-7492t\") pod \"payload-processing-e2e-ait-6abd1847-5c7544d97d-jnvql\" (UID: \"4c5b1b12-fd28-4b1b-9440-525eaca94984\") " pod="openshift-ingress/payload-processing-e2e-ait-6abd1847-5c7544d97d-jnvql"
Jul 26 16:17:23.337101 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:23.336965    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79de5b67-3416-40b1-baf6-2ea1f5599631-maas-api-tls\") pod \"maas-api-e2e-ait-6abd1847-58b7c9b654-r849s\" (UID: \"79de5b67-3416-40b1-baf6-2ea1f5599631\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6abd1847-58b7c9b654-r849s"
Jul 26 16:17:23.339422 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:23.339393    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79de5b67-3416-40b1-baf6-2ea1f5599631-maas-api-tls\") pod \"maas-api-e2e-ait-6abd1847-58b7c9b654-r849s\" (UID: \"79de5b67-3416-40b1-baf6-2ea1f5599631\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6abd1847-58b7c9b654-r849s"
Jul 26 16:17:23.437985 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:23.437900    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c61a6df-8f37-477b-967e-bb400bdde08c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6abd1847-8c9cc4b8-vx2gx\" (UID: \"7c61a6df-8f37-477b-967e-bb400bdde08c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6abd1847-8c9cc4b8-vx2gx"
Jul 26 16:17:23.437985 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:23.437939    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7492t\" (UniqueName: \"kubernetes.io/projected/4c5b1b12-fd28-4b1b-9440-525eaca94984-kube-api-access-7492t\") pod \"payload-processing-e2e-ait-6abd1847-5c7544d97d-jnvql\" (UID: \"4c5b1b12-fd28-4b1b-9440-525eaca94984\") " pod="openshift-ingress/payload-processing-e2e-ait-6abd1847-5c7544d97d-jnvql"
Jul 26 16:17:23.438206 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:23.437992    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c5b1b12-fd28-4b1b-9440-525eaca94984-plugins-config-volume\") pod \"payload-processing-e2e-ait-6abd1847-5c7544d97d-jnvql\" (UID: \"4c5b1b12-fd28-4b1b-9440-525eaca94984\") " pod="openshift-ingress/payload-processing-e2e-ait-6abd1847-5c7544d97d-jnvql"
Jul 26 16:17:23.438206 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:23.438027    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnlcd\" (UniqueName: \"kubernetes.io/projected/7c61a6df-8f37-477b-967e-bb400bdde08c-kube-api-access-pnlcd\") pod \"payload-pre-processing-e2e-ait-6abd1847-8c9cc4b8-vx2gx\" (UID: \"7c61a6df-8f37-477b-967e-bb400bdde08c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6abd1847-8c9cc4b8-vx2gx"
Jul 26 16:17:23.438849 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:23.438826    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c5b1b12-fd28-4b1b-9440-525eaca94984-plugins-config-volume\") pod \"payload-processing-e2e-ait-6abd1847-5c7544d97d-jnvql\" (UID: \"4c5b1b12-fd28-4b1b-9440-525eaca94984\") " pod="openshift-ingress/payload-processing-e2e-ait-6abd1847-5c7544d97d-jnvql"
Jul 26 16:17:23.438849 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:23.438841    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c61a6df-8f37-477b-967e-bb400bdde08c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6abd1847-8c9cc4b8-vx2gx\" (UID: \"7c61a6df-8f37-477b-967e-bb400bdde08c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6abd1847-8c9cc4b8-vx2gx"
Jul 26 16:17:23.446113 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:23.446083    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7492t\" (UniqueName: \"kubernetes.io/projected/4c5b1b12-fd28-4b1b-9440-525eaca94984-kube-api-access-7492t\") pod \"payload-processing-e2e-ait-6abd1847-5c7544d97d-jnvql\" (UID: \"4c5b1b12-fd28-4b1b-9440-525eaca94984\") " pod="openshift-ingress/payload-processing-e2e-ait-6abd1847-5c7544d97d-jnvql"
Jul 26 16:17:23.446536 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:23.446517    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnlcd\" (UniqueName: \"kubernetes.io/projected/7c61a6df-8f37-477b-967e-bb400bdde08c-kube-api-access-pnlcd\") pod \"payload-pre-processing-e2e-ait-6abd1847-8c9cc4b8-vx2gx\" (UID: \"7c61a6df-8f37-477b-967e-bb400bdde08c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6abd1847-8c9cc4b8-vx2gx"
Jul 26 16:17:23.476048 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:23.476016    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6abd1847-58b7c9b654-r849s"
Jul 26 16:17:23.486733 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:23.486293    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6abd1847-8c9cc4b8-vx2gx"
Jul 26 16:17:23.506231 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:23.506200    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6abd1847-5c7544d97d-jnvql"
Jul 26 16:17:23.650421 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:23.650391    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6abd1847-58b7c9b654-r849s"]
Jul 26 16:17:23.653193 ip-10-0-133-204 kubenswrapper[2563]: W0726 16:17:23.653150    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79de5b67_3416_40b1_baf6_2ea1f5599631.slice/crio-cd71de0368bb704817f6208693015b9d7a88187b6cbcb1eb47cc543dfec69b21 WatchSource:0}: Error finding container cd71de0368bb704817f6208693015b9d7a88187b6cbcb1eb47cc543dfec69b21: Status 404 returned error can't find the container with id cd71de0368bb704817f6208693015b9d7a88187b6cbcb1eb47cc543dfec69b21
Jul 26 16:17:23.655298 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:23.655278    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 26 16:17:23.671129 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:23.671096    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6abd1847-8c9cc4b8-vx2gx"]
Jul 26 16:17:23.673730 ip-10-0-133-204 kubenswrapper[2563]: W0726 16:17:23.673144    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c61a6df_8f37_477b_967e_bb400bdde08c.slice/crio-6680bfd99c606f21c7c60d646f15c4d1b494ec8e177bd723b63feba935180b00 WatchSource:0}: Error finding container 6680bfd99c606f21c7c60d646f15c4d1b494ec8e177bd723b63feba935180b00: Status 404 returned error can't find the container with id 6680bfd99c606f21c7c60d646f15c4d1b494ec8e177bd723b63feba935180b00
Jul 26 16:17:23.698984 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:23.698960    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6abd1847-5c7544d97d-jnvql"]
Jul 26 16:17:23.700930 ip-10-0-133-204 kubenswrapper[2563]: W0726 16:17:23.700906    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c5b1b12_fd28_4b1b_9440_525eaca94984.slice/crio-827b818cd448406bfb4109fa4aafa0247438c3990ac1656765f5befe8eb9107d WatchSource:0}: Error finding container 827b818cd448406bfb4109fa4aafa0247438c3990ac1656765f5befe8eb9107d: Status 404 returned error can't find the container with id 827b818cd448406bfb4109fa4aafa0247438c3990ac1656765f5befe8eb9107d
Jul 26 16:17:24.239357 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:24.239302    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6abd1847-8c9cc4b8-vx2gx" event={"ID":"7c61a6df-8f37-477b-967e-bb400bdde08c","Type":"ContainerStarted","Data":"6e6cb21c9dfeacb861c98d4bf9b24c8c5fb8060af9e5c7277e0853124438a3f2"}
Jul 26 16:17:24.239539 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:24.239409    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-6abd1847-8c9cc4b8-vx2gx"
Jul 26 16:17:24.239539 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:24.239433    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6abd1847-8c9cc4b8-vx2gx" event={"ID":"7c61a6df-8f37-477b-967e-bb400bdde08c","Type":"ContainerStarted","Data":"6680bfd99c606f21c7c60d646f15c4d1b494ec8e177bd723b63feba935180b00"}
Jul 26 16:17:24.240875 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:24.240853    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6abd1847-5c7544d97d-jnvql" event={"ID":"4c5b1b12-fd28-4b1b-9440-525eaca94984","Type":"ContainerStarted","Data":"4ad3cbcbb06ffba3770a107adf0bbda12ceeecf935c4ed21ffc9d4f0038ca2d5"}
Jul 26 16:17:24.240943 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:24.240897    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6abd1847-5c7544d97d-jnvql" event={"ID":"4c5b1b12-fd28-4b1b-9440-525eaca94984","Type":"ContainerStarted","Data":"827b818cd448406bfb4109fa4aafa0247438c3990ac1656765f5befe8eb9107d"}
Jul 26 16:17:24.240943 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:24.240926    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-6abd1847-5c7544d97d-jnvql"
Jul 26 16:17:24.242309 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:24.242285    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6abd1847-58b7c9b654-r849s" event={"ID":"79de5b67-3416-40b1-baf6-2ea1f5599631","Type":"ContainerStarted","Data":"b896a0983d6b9205359354a34781b4cdabad1def9789b61e5f3df871dfc14448"}
Jul 26 16:17:24.242444 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:24.242321    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6abd1847-58b7c9b654-r849s" event={"ID":"79de5b67-3416-40b1-baf6-2ea1f5599631","Type":"ContainerStarted","Data":"cd71de0368bb704817f6208693015b9d7a88187b6cbcb1eb47cc543dfec69b21"}
Jul 26 16:17:24.242444 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:24.242417    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6abd1847-58b7c9b654-r849s"
Jul 26 16:17:24.256109 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:24.256062    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-6abd1847-8c9cc4b8-vx2gx" podStartSLOduration=1.2560475420000001 podStartE2EDuration="1.256047542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:17:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:17:24.254276602 +0000 UTC m=+1636.358502429" watchObservedRunningTime="2026-07-26 16:17:24.256047542 +0000 UTC m=+1636.360273351"
Jul 26 16:17:24.272546 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:24.272492    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6abd1847-58b7c9b654-r849s" podStartSLOduration=1.877564026 podStartE2EDuration="2.272473945s" totalImagesPullingTime="394.909919ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:17:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:17:24.270918602 +0000 UTC m=+1636.375144444" watchObservedRunningTime="2026-07-26 16:17:24.272473945 +0000 UTC m=+1636.376699756"
Jul 26 16:17:24.289751 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:24.289691    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-6abd1847-5c7544d97d-jnvql" podStartSLOduration=1.289669077 podStartE2EDuration="1.289669077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:17:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:17:24.287852461 +0000 UTC m=+1636.392078283" watchObservedRunningTime="2026-07-26 16:17:24.289669077 +0000 UTC m=+1636.393894887"
Jul 26 16:17:30.253860 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:30.253826    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6abd1847-58b7c9b654-r849s"
Jul 26 16:17:35.249971 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:35.249941    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-6abd1847-8c9cc4b8-vx2gx"
Jul 26 16:17:35.250374 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:35.250022    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-6abd1847-5c7544d97d-jnvql"
Jul 26 16:17:58.399453 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:58.399415    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6abd1847-58b7c9b654-r849s"]
Jul 26 16:17:58.400106 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:58.399710    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6abd1847-58b7c9b654-r849s" podUID="79de5b67-3416-40b1-baf6-2ea1f5599631" containerName="maas-api" containerID="cri-o://b896a0983d6b9205359354a34781b4cdabad1def9789b61e5f3df871dfc14448" gracePeriod=30
Jul 26 16:17:58.431410 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:58.431378    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:17:58.459504 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:58.459470    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6abd1847-5c7544d97d-jnvql"]
Jul 26 16:17:58.459708 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:58.459671    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-6abd1847-5c7544d97d-jnvql" podUID="4c5b1b12-fd28-4b1b-9440-525eaca94984" containerName="payload-processing" containerID="cri-o://4ad3cbcbb06ffba3770a107adf0bbda12ceeecf935c4ed21ffc9d4f0038ca2d5" gracePeriod=30
Jul 26 16:17:58.466906 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:58.466555    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6abd1847-8c9cc4b8-vx2gx"]
Jul 26 16:17:58.467399 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:58.467359    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-6abd1847-8c9cc4b8-vx2gx" podUID="7c61a6df-8f37-477b-967e-bb400bdde08c" containerName="payload-pre-processing" containerID="cri-o://6e6cb21c9dfeacb861c98d4bf9b24c8c5fb8060af9e5c7277e0853124438a3f2" gracePeriod=30
Jul 26 16:17:58.551734 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:58.551700    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:17:58.667673 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:58.667454    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:17:58.731537 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:58.731510    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6abd1847-58b7c9b654-r849s"
Jul 26 16:17:58.771850 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:58.771820    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6abd1847-8c9cc4b8-vx2gx"
Jul 26 16:17:58.777457 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:58.777438    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6abd1847-5c7544d97d-jnvql"
Jul 26 16:17:58.871287 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:58.871257    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79de5b67-3416-40b1-baf6-2ea1f5599631-maas-api-tls\") pod \"79de5b67-3416-40b1-baf6-2ea1f5599631\" (UID: \"79de5b67-3416-40b1-baf6-2ea1f5599631\") "
Jul 26 16:17:58.871485 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:58.871300    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79de5b67-3416-40b1-baf6-2ea1f5599631-openshift-service-ca\") pod \"79de5b67-3416-40b1-baf6-2ea1f5599631\" (UID: \"79de5b67-3416-40b1-baf6-2ea1f5599631\") "
Jul 26 16:17:58.871485 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:58.871335    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tnsqh\" (UniqueName: \"kubernetes.io/projected/79de5b67-3416-40b1-baf6-2ea1f5599631-kube-api-access-tnsqh\") pod \"79de5b67-3416-40b1-baf6-2ea1f5599631\" (UID: \"79de5b67-3416-40b1-baf6-2ea1f5599631\") "
Jul 26 16:17:58.871485 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:58.871379    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c5b1b12-fd28-4b1b-9440-525eaca94984-plugins-config-volume\") pod \"4c5b1b12-fd28-4b1b-9440-525eaca94984\" (UID: \"4c5b1b12-fd28-4b1b-9440-525eaca94984\") "
Jul 26 16:17:58.871485 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:58.871408    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pnlcd\" (UniqueName: \"kubernetes.io/projected/7c61a6df-8f37-477b-967e-bb400bdde08c-kube-api-access-pnlcd\") pod \"7c61a6df-8f37-477b-967e-bb400bdde08c\" (UID: \"7c61a6df-8f37-477b-967e-bb400bdde08c\") "
Jul 26 16:17:58.871485 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:58.871433    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7492t\" (UniqueName: \"kubernetes.io/projected/4c5b1b12-fd28-4b1b-9440-525eaca94984-kube-api-access-7492t\") pod \"4c5b1b12-fd28-4b1b-9440-525eaca94984\" (UID: \"4c5b1b12-fd28-4b1b-9440-525eaca94984\") "
Jul 26 16:17:58.871789 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:58.871552    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c61a6df-8f37-477b-967e-bb400bdde08c-plugins-config-volume\") pod \"7c61a6df-8f37-477b-967e-bb400bdde08c\" (UID: \"7c61a6df-8f37-477b-967e-bb400bdde08c\") "
Jul 26 16:17:58.871789 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:58.871702    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79de5b67-3416-40b1-baf6-2ea1f5599631-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "79de5b67-3416-40b1-baf6-2ea1f5599631" (UID: "79de5b67-3416-40b1-baf6-2ea1f5599631"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 26 16:17:58.871906 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:58.871810    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c5b1b12-fd28-4b1b-9440-525eaca94984-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4c5b1b12-fd28-4b1b-9440-525eaca94984" (UID: "4c5b1b12-fd28-4b1b-9440-525eaca94984"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 26 16:17:58.872116 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:58.872052    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c61a6df-8f37-477b-967e-bb400bdde08c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7c61a6df-8f37-477b-967e-bb400bdde08c" (UID: "7c61a6df-8f37-477b-967e-bb400bdde08c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 26 16:17:58.872116 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:58.872057    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79de5b67-3416-40b1-baf6-2ea1f5599631-openshift-service-ca\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:17:58.872116 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:58.872094    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c5b1b12-fd28-4b1b-9440-525eaca94984-plugins-config-volume\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:17:58.873863 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:58.873839    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79de5b67-3416-40b1-baf6-2ea1f5599631-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "79de5b67-3416-40b1-baf6-2ea1f5599631" (UID: "79de5b67-3416-40b1-baf6-2ea1f5599631"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 26 16:17:58.874018 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:58.873993    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79de5b67-3416-40b1-baf6-2ea1f5599631-kube-api-access-tnsqh" (OuterVolumeSpecName: "kube-api-access-tnsqh") pod "79de5b67-3416-40b1-baf6-2ea1f5599631" (UID: "79de5b67-3416-40b1-baf6-2ea1f5599631"). InnerVolumeSpecName "kube-api-access-tnsqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 26 16:17:58.874305 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:58.874280    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c61a6df-8f37-477b-967e-bb400bdde08c-kube-api-access-pnlcd" (OuterVolumeSpecName: "kube-api-access-pnlcd") pod "7c61a6df-8f37-477b-967e-bb400bdde08c" (UID: "7c61a6df-8f37-477b-967e-bb400bdde08c"). InnerVolumeSpecName "kube-api-access-pnlcd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 26 16:17:58.874362 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:58.874316    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c5b1b12-fd28-4b1b-9440-525eaca94984-kube-api-access-7492t" (OuterVolumeSpecName: "kube-api-access-7492t") pod "4c5b1b12-fd28-4b1b-9440-525eaca94984" (UID: "4c5b1b12-fd28-4b1b-9440-525eaca94984"). InnerVolumeSpecName "kube-api-access-7492t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 26 16:17:58.973151 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:58.973117    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c61a6df-8f37-477b-967e-bb400bdde08c-plugins-config-volume\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:17:58.973151 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:58.973150    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79de5b67-3416-40b1-baf6-2ea1f5599631-maas-api-tls\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:17:58.973363 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:58.973163    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnsqh\" (UniqueName: \"kubernetes.io/projected/79de5b67-3416-40b1-baf6-2ea1f5599631-kube-api-access-tnsqh\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:17:58.973363 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:58.973174    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnlcd\" (UniqueName: \"kubernetes.io/projected/7c61a6df-8f37-477b-967e-bb400bdde08c-kube-api-access-pnlcd\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:17:58.973363 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:58.973187    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7492t\" (UniqueName: \"kubernetes.io/projected/4c5b1b12-fd28-4b1b-9440-525eaca94984-kube-api-access-7492t\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:17:59.418105 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:59.418072    2563 generic.go:356] "Generic (PLEG): container finished" podID="4c5b1b12-fd28-4b1b-9440-525eaca94984" containerID="4ad3cbcbb06ffba3770a107adf0bbda12ceeecf935c4ed21ffc9d4f0038ca2d5" exitCode=0
Jul 26 16:17:59.418597 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:59.418137    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6abd1847-5c7544d97d-jnvql"
Jul 26 16:17:59.418597 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:59.418148    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6abd1847-5c7544d97d-jnvql" event={"ID":"4c5b1b12-fd28-4b1b-9440-525eaca94984","Type":"ContainerDied","Data":"4ad3cbcbb06ffba3770a107adf0bbda12ceeecf935c4ed21ffc9d4f0038ca2d5"}
Jul 26 16:17:59.418597 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:59.418222    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6abd1847-5c7544d97d-jnvql" event={"ID":"4c5b1b12-fd28-4b1b-9440-525eaca94984","Type":"ContainerDied","Data":"827b818cd448406bfb4109fa4aafa0247438c3990ac1656765f5befe8eb9107d"}
Jul 26 16:17:59.418597 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:59.418256    2563 scope.go:117] "RemoveContainer" containerID="4ad3cbcbb06ffba3770a107adf0bbda12ceeecf935c4ed21ffc9d4f0038ca2d5"
Jul 26 16:17:59.420651 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:59.420282    2563 generic.go:356] "Generic (PLEG): container finished" podID="79de5b67-3416-40b1-baf6-2ea1f5599631" containerID="b896a0983d6b9205359354a34781b4cdabad1def9789b61e5f3df871dfc14448" exitCode=0
Jul 26 16:17:59.420651 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:59.420366    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6abd1847-58b7c9b654-r849s"
Jul 26 16:17:59.420651 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:59.420375    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6abd1847-58b7c9b654-r849s" event={"ID":"79de5b67-3416-40b1-baf6-2ea1f5599631","Type":"ContainerDied","Data":"b896a0983d6b9205359354a34781b4cdabad1def9789b61e5f3df871dfc14448"}
Jul 26 16:17:59.420651 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:59.420480    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6abd1847-58b7c9b654-r849s" event={"ID":"79de5b67-3416-40b1-baf6-2ea1f5599631","Type":"ContainerDied","Data":"cd71de0368bb704817f6208693015b9d7a88187b6cbcb1eb47cc543dfec69b21"}
Jul 26 16:17:59.422414 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:59.422389    2563 generic.go:356] "Generic (PLEG): container finished" podID="7c61a6df-8f37-477b-967e-bb400bdde08c" containerID="6e6cb21c9dfeacb861c98d4bf9b24c8c5fb8060af9e5c7277e0853124438a3f2" exitCode=0
Jul 26 16:17:59.422536 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:59.422442    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6abd1847-8c9cc4b8-vx2gx"
Jul 26 16:17:59.422536 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:59.422479    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6abd1847-8c9cc4b8-vx2gx" event={"ID":"7c61a6df-8f37-477b-967e-bb400bdde08c","Type":"ContainerDied","Data":"6e6cb21c9dfeacb861c98d4bf9b24c8c5fb8060af9e5c7277e0853124438a3f2"}
Jul 26 16:17:59.422536 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:59.422520    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6abd1847-8c9cc4b8-vx2gx" event={"ID":"7c61a6df-8f37-477b-967e-bb400bdde08c","Type":"ContainerDied","Data":"6680bfd99c606f21c7c60d646f15c4d1b494ec8e177bd723b63feba935180b00"}
Jul 26 16:17:59.432532 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:59.432512    2563 scope.go:117] "RemoveContainer" containerID="4ad3cbcbb06ffba3770a107adf0bbda12ceeecf935c4ed21ffc9d4f0038ca2d5"
Jul 26 16:17:59.432866 ip-10-0-133-204 kubenswrapper[2563]: E0726 16:17:59.432841    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ad3cbcbb06ffba3770a107adf0bbda12ceeecf935c4ed21ffc9d4f0038ca2d5\": container with ID starting with 4ad3cbcbb06ffba3770a107adf0bbda12ceeecf935c4ed21ffc9d4f0038ca2d5 not found: ID does not exist" containerID="4ad3cbcbb06ffba3770a107adf0bbda12ceeecf935c4ed21ffc9d4f0038ca2d5"
Jul 26 16:17:59.432948 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:59.432877    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ad3cbcbb06ffba3770a107adf0bbda12ceeecf935c4ed21ffc9d4f0038ca2d5"} err="failed to get container status \"4ad3cbcbb06ffba3770a107adf0bbda12ceeecf935c4ed21ffc9d4f0038ca2d5\": rpc error: code = NotFound desc = could not find container \"4ad3cbcbb06ffba3770a107adf0bbda12ceeecf935c4ed21ffc9d4f0038ca2d5\": container with ID starting with 4ad3cbcbb06ffba3770a107adf0bbda12ceeecf935c4ed21ffc9d4f0038ca2d5 not found: ID does not exist"
Jul 26 16:17:59.432948 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:59.432896    2563 scope.go:117] "RemoveContainer" containerID="b896a0983d6b9205359354a34781b4cdabad1def9789b61e5f3df871dfc14448"
Jul 26 16:17:59.442924 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:59.442903    2563 scope.go:117] "RemoveContainer" containerID="b896a0983d6b9205359354a34781b4cdabad1def9789b61e5f3df871dfc14448"
Jul 26 16:17:59.443182 ip-10-0-133-204 kubenswrapper[2563]: E0726 16:17:59.443164    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b896a0983d6b9205359354a34781b4cdabad1def9789b61e5f3df871dfc14448\": container with ID starting with b896a0983d6b9205359354a34781b4cdabad1def9789b61e5f3df871dfc14448 not found: ID does not exist" containerID="b896a0983d6b9205359354a34781b4cdabad1def9789b61e5f3df871dfc14448"
Jul 26 16:17:59.443246 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:59.443187    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b896a0983d6b9205359354a34781b4cdabad1def9789b61e5f3df871dfc14448"} err="failed to get container status \"b896a0983d6b9205359354a34781b4cdabad1def9789b61e5f3df871dfc14448\": rpc error: code = NotFound desc = could not find container \"b896a0983d6b9205359354a34781b4cdabad1def9789b61e5f3df871dfc14448\": container with ID starting with b896a0983d6b9205359354a34781b4cdabad1def9789b61e5f3df871dfc14448 not found: ID does not exist"
Jul 26 16:17:59.443246 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:59.443204    2563 scope.go:117] "RemoveContainer" containerID="6e6cb21c9dfeacb861c98d4bf9b24c8c5fb8060af9e5c7277e0853124438a3f2"
Jul 26 16:17:59.444532 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:59.444509    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6abd1847-5c7544d97d-jnvql"]
Jul 26 16:17:59.448905 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:59.448884    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6abd1847-5c7544d97d-jnvql"]
Jul 26 16:17:59.451940 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:59.451922    2563 scope.go:117] "RemoveContainer" containerID="6e6cb21c9dfeacb861c98d4bf9b24c8c5fb8060af9e5c7277e0853124438a3f2"
Jul 26 16:17:59.452190 ip-10-0-133-204 kubenswrapper[2563]: E0726 16:17:59.452174    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e6cb21c9dfeacb861c98d4bf9b24c8c5fb8060af9e5c7277e0853124438a3f2\": container with ID starting with 6e6cb21c9dfeacb861c98d4bf9b24c8c5fb8060af9e5c7277e0853124438a3f2 not found: ID does not exist" containerID="6e6cb21c9dfeacb861c98d4bf9b24c8c5fb8060af9e5c7277e0853124438a3f2"
Jul 26 16:17:59.452243 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:59.452196    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e6cb21c9dfeacb861c98d4bf9b24c8c5fb8060af9e5c7277e0853124438a3f2"} err="failed to get container status \"6e6cb21c9dfeacb861c98d4bf9b24c8c5fb8060af9e5c7277e0853124438a3f2\": rpc error: code = NotFound desc = could not find container \"6e6cb21c9dfeacb861c98d4bf9b24c8c5fb8060af9e5c7277e0853124438a3f2\": container with ID starting with 6e6cb21c9dfeacb861c98d4bf9b24c8c5fb8060af9e5c7277e0853124438a3f2 not found: ID does not exist"
Jul 26 16:17:59.459019 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:59.458998    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6abd1847-8c9cc4b8-vx2gx"]
Jul 26 16:17:59.462796 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:59.462755    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6abd1847-8c9cc4b8-vx2gx"]
Jul 26 16:17:59.473356 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:59.473334    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6abd1847-58b7c9b654-r849s"]
Jul 26 16:17:59.479472 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:17:59.479452    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6abd1847-58b7c9b654-r849s"]
Jul 26 16:18:00.373429 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:00.373394    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c5b1b12-fd28-4b1b-9440-525eaca94984" path="/var/lib/kubelet/pods/4c5b1b12-fd28-4b1b-9440-525eaca94984/volumes"
Jul 26 16:18:00.373831 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:00.373817    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79de5b67-3416-40b1-baf6-2ea1f5599631" path="/var/lib/kubelet/pods/79de5b67-3416-40b1-baf6-2ea1f5599631/volumes"
Jul 26 16:18:00.374134 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:00.374123    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c61a6df-8f37-477b-967e-bb400bdde08c" path="/var/lib/kubelet/pods/7c61a6df-8f37-477b-967e-bb400bdde08c/volumes"
Jul 26 16:18:03.040985 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:03.040940    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:18:03.201085 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:03.201051    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:18:03.449063 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:03.448983    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:18:03.725081 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:03.725047    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:18:03.833026 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:03.832986    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:18:03.941446 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:03.941409    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:18:04.266445 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:04.266397    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:18:04.597166 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:04.597079    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:18:06.671075 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:06.671036    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:18:06.939392 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:06.939301    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d78a04d0-669d6d6fc4-6b8k9"]
Jul 26 16:18:06.939703 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:06.939691    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c5b1b12-fd28-4b1b-9440-525eaca94984" containerName="payload-processing"
Jul 26 16:18:06.939776 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:06.939704    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c5b1b12-fd28-4b1b-9440-525eaca94984" containerName="payload-processing"
Jul 26 16:18:06.939776 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:06.939717    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79de5b67-3416-40b1-baf6-2ea1f5599631" containerName="maas-api"
Jul 26 16:18:06.939776 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:06.939723    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="79de5b67-3416-40b1-baf6-2ea1f5599631" containerName="maas-api"
Jul 26 16:18:06.939776 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:06.939737    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c61a6df-8f37-477b-967e-bb400bdde08c" containerName="payload-pre-processing"
Jul 26 16:18:06.939776 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:06.939743    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c61a6df-8f37-477b-967e-bb400bdde08c" containerName="payload-pre-processing"
Jul 26 16:18:06.940057 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:06.939829    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c61a6df-8f37-477b-967e-bb400bdde08c" containerName="payload-pre-processing"
Jul 26 16:18:06.940057 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:06.939836    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c5b1b12-fd28-4b1b-9440-525eaca94984" containerName="payload-processing"
Jul 26 16:18:06.940057 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:06.939843    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="79de5b67-3416-40b1-baf6-2ea1f5599631" containerName="maas-api"
Jul 26 16:18:06.944642 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:06.944619    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d78a04d0-669d6d6fc4-6b8k9"
Jul 26 16:18:06.946947 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:06.946928    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-d78a04d0-serving-cert\""
Jul 26 16:18:06.956262 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:06.956234    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d78a04d0-669d6d6fc4-6b8k9"]
Jul 26 16:18:06.978037 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:06.978002    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:18:07.049968 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:07.049936    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e17a5df2-c4f1-441d-af7a-6887f7db1c5e-openshift-service-ca\") pod \"maas-api-e2e-migrate-d78a04d0-669d6d6fc4-6b8k9\" (UID: \"e17a5df2-c4f1-441d-af7a-6887f7db1c5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d78a04d0-669d6d6fc4-6b8k9"
Jul 26 16:18:07.050146 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:07.050045    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e17a5df2-c4f1-441d-af7a-6887f7db1c5e-maas-api-tls\") pod \"maas-api-e2e-migrate-d78a04d0-669d6d6fc4-6b8k9\" (UID: \"e17a5df2-c4f1-441d-af7a-6887f7db1c5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d78a04d0-669d6d6fc4-6b8k9"
Jul 26 16:18:07.050146 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:07.050092    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktv8k\" (UniqueName: \"kubernetes.io/projected/e17a5df2-c4f1-441d-af7a-6887f7db1c5e-kube-api-access-ktv8k\") pod \"maas-api-e2e-migrate-d78a04d0-669d6d6fc4-6b8k9\" (UID: \"e17a5df2-c4f1-441d-af7a-6887f7db1c5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d78a04d0-669d6d6fc4-6b8k9"
Jul 26 16:18:07.150675 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:07.150637    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e17a5df2-c4f1-441d-af7a-6887f7db1c5e-maas-api-tls\") pod \"maas-api-e2e-migrate-d78a04d0-669d6d6fc4-6b8k9\" (UID: \"e17a5df2-c4f1-441d-af7a-6887f7db1c5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d78a04d0-669d6d6fc4-6b8k9"
Jul 26 16:18:07.150901 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:07.150699    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktv8k\" (UniqueName: \"kubernetes.io/projected/e17a5df2-c4f1-441d-af7a-6887f7db1c5e-kube-api-access-ktv8k\") pod \"maas-api-e2e-migrate-d78a04d0-669d6d6fc4-6b8k9\" (UID: \"e17a5df2-c4f1-441d-af7a-6887f7db1c5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d78a04d0-669d6d6fc4-6b8k9"
Jul 26 16:18:07.150901 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:07.150728    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e17a5df2-c4f1-441d-af7a-6887f7db1c5e-openshift-service-ca\") pod \"maas-api-e2e-migrate-d78a04d0-669d6d6fc4-6b8k9\" (UID: \"e17a5df2-c4f1-441d-af7a-6887f7db1c5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d78a04d0-669d6d6fc4-6b8k9"
Jul 26 16:18:07.150901 ip-10-0-133-204 kubenswrapper[2563]: E0726 16:18:07.150812    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-d78a04d0-serving-cert: secret "maas-api-e2e-migrate-d78a04d0-serving-cert" not found
Jul 26 16:18:07.150901 ip-10-0-133-204 kubenswrapper[2563]: E0726 16:18:07.150873    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e17a5df2-c4f1-441d-af7a-6887f7db1c5e-maas-api-tls podName:e17a5df2-c4f1-441d-af7a-6887f7db1c5e nodeName:}" failed. No retries permitted until 2026-07-26 16:18:07.650854348 +0000 UTC m=+1679.755080135 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e17a5df2-c4f1-441d-af7a-6887f7db1c5e-maas-api-tls") pod "maas-api-e2e-migrate-d78a04d0-669d6d6fc4-6b8k9" (UID: "e17a5df2-c4f1-441d-af7a-6887f7db1c5e") : secret "maas-api-e2e-migrate-d78a04d0-serving-cert" not found
Jul 26 16:18:07.151419 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:07.151399    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e17a5df2-c4f1-441d-af7a-6887f7db1c5e-openshift-service-ca\") pod \"maas-api-e2e-migrate-d78a04d0-669d6d6fc4-6b8k9\" (UID: \"e17a5df2-c4f1-441d-af7a-6887f7db1c5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d78a04d0-669d6d6fc4-6b8k9"
Jul 26 16:18:07.158872 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:07.158848    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktv8k\" (UniqueName: \"kubernetes.io/projected/e17a5df2-c4f1-441d-af7a-6887f7db1c5e-kube-api-access-ktv8k\") pod \"maas-api-e2e-migrate-d78a04d0-669d6d6fc4-6b8k9\" (UID: \"e17a5df2-c4f1-441d-af7a-6887f7db1c5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d78a04d0-669d6d6fc4-6b8k9"
Jul 26 16:18:07.442213 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:07.442178    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:18:07.656315 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:07.656276    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e17a5df2-c4f1-441d-af7a-6887f7db1c5e-maas-api-tls\") pod \"maas-api-e2e-migrate-d78a04d0-669d6d6fc4-6b8k9\" (UID: \"e17a5df2-c4f1-441d-af7a-6887f7db1c5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d78a04d0-669d6d6fc4-6b8k9"
Jul 26 16:18:07.658728 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:07.658701    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e17a5df2-c4f1-441d-af7a-6887f7db1c5e-maas-api-tls\") pod \"maas-api-e2e-migrate-d78a04d0-669d6d6fc4-6b8k9\" (UID: \"e17a5df2-c4f1-441d-af7a-6887f7db1c5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d78a04d0-669d6d6fc4-6b8k9"
Jul 26 16:18:07.675404 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:07.675377    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:18:07.777907 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:07.777873    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:18:07.788099 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:07.788068    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d78a04d0-6bc8f8549b-cdqdc"]
Jul 26 16:18:07.791809 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:07.791786    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d78a04d0-6bc8f8549b-cdqdc"
Jul 26 16:18:07.794323 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:07.794302    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-d78a04d0\""
Jul 26 16:18:07.794457 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:07.794301    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-d78a04d0-dockercfg-4lhsl\""
Jul 26 16:18:07.812493 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:07.812451    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d78a04d0-6bc8f8549b-cdqdc"]
Jul 26 16:18:07.836529 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:07.836493    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d78a04d0-8479c4d7f7-j94px"]
Jul 26 16:18:07.839920 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:07.839902    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d78a04d0-8479c4d7f7-j94px"
Jul 26 16:18:07.855291 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:07.855260    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d78a04d0-8479c4d7f7-j94px"]
Jul 26 16:18:07.857667 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:07.857642    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d78a04d0-669d6d6fc4-6b8k9"
Jul 26 16:18:07.958217 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:07.958151    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxt48\" (UniqueName: \"kubernetes.io/projected/e0a4cf62-6b8e-4e88-8c36-85597c9f2638-kube-api-access-cxt48\") pod \"payload-pre-processing-e2e-migrate-d78a04d0-6bc8f8549b-cdqdc\" (UID: \"e0a4cf62-6b8e-4e88-8c36-85597c9f2638\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d78a04d0-6bc8f8549b-cdqdc"
Jul 26 16:18:07.958415 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:07.958377    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktnjn\" (UniqueName: \"kubernetes.io/projected/dff03dcd-30c4-41ee-9cfb-83e9d3a8c4fb-kube-api-access-ktnjn\") pod \"payload-processing-e2e-migrate-d78a04d0-8479c4d7f7-j94px\" (UID: \"dff03dcd-30c4-41ee-9cfb-83e9d3a8c4fb\") " pod="openshift-ingress/payload-processing-e2e-migrate-d78a04d0-8479c4d7f7-j94px"
Jul 26 16:18:07.958500 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:07.958459    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0a4cf62-6b8e-4e88-8c36-85597c9f2638-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-d78a04d0-6bc8f8549b-cdqdc\" (UID: \"e0a4cf62-6b8e-4e88-8c36-85597c9f2638\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d78a04d0-6bc8f8549b-cdqdc"
Jul 26 16:18:07.958568 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:07.958529    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dff03dcd-30c4-41ee-9cfb-83e9d3a8c4fb-plugins-config-volume\") pod \"payload-processing-e2e-migrate-d78a04d0-8479c4d7f7-j94px\" (UID: \"dff03dcd-30c4-41ee-9cfb-83e9d3a8c4fb\") " pod="openshift-ingress/payload-processing-e2e-migrate-d78a04d0-8479c4d7f7-j94px"
Jul 26 16:18:08.006154 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:08.006132    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d78a04d0-669d6d6fc4-6b8k9"]
Jul 26 16:18:08.008869 ip-10-0-133-204 kubenswrapper[2563]: W0726 16:18:08.008839    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode17a5df2_c4f1_441d_af7a_6887f7db1c5e.slice/crio-0cdcb866f23a9a0449ec42aea5cd2b04c40708d8eac1ec80822e1f8a6cc76a54 WatchSource:0}: Error finding container 0cdcb866f23a9a0449ec42aea5cd2b04c40708d8eac1ec80822e1f8a6cc76a54: Status 404 returned error can't find the container with id 0cdcb866f23a9a0449ec42aea5cd2b04c40708d8eac1ec80822e1f8a6cc76a54
Jul 26 16:18:08.059169 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:08.059079    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxt48\" (UniqueName: \"kubernetes.io/projected/e0a4cf62-6b8e-4e88-8c36-85597c9f2638-kube-api-access-cxt48\") pod \"payload-pre-processing-e2e-migrate-d78a04d0-6bc8f8549b-cdqdc\" (UID: \"e0a4cf62-6b8e-4e88-8c36-85597c9f2638\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d78a04d0-6bc8f8549b-cdqdc"
Jul 26 16:18:08.059169 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:08.059134    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktnjn\" (UniqueName: \"kubernetes.io/projected/dff03dcd-30c4-41ee-9cfb-83e9d3a8c4fb-kube-api-access-ktnjn\") pod \"payload-processing-e2e-migrate-d78a04d0-8479c4d7f7-j94px\" (UID: \"dff03dcd-30c4-41ee-9cfb-83e9d3a8c4fb\") " pod="openshift-ingress/payload-processing-e2e-migrate-d78a04d0-8479c4d7f7-j94px"
Jul 26 16:18:08.059357 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:08.059175    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0a4cf62-6b8e-4e88-8c36-85597c9f2638-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-d78a04d0-6bc8f8549b-cdqdc\" (UID: \"e0a4cf62-6b8e-4e88-8c36-85597c9f2638\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d78a04d0-6bc8f8549b-cdqdc"
Jul 26 16:18:08.059357 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:08.059208    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dff03dcd-30c4-41ee-9cfb-83e9d3a8c4fb-plugins-config-volume\") pod \"payload-processing-e2e-migrate-d78a04d0-8479c4d7f7-j94px\" (UID: \"dff03dcd-30c4-41ee-9cfb-83e9d3a8c4fb\") " pod="openshift-ingress/payload-processing-e2e-migrate-d78a04d0-8479c4d7f7-j94px"
Jul 26 16:18:08.060022 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:08.059999    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0a4cf62-6b8e-4e88-8c36-85597c9f2638-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-d78a04d0-6bc8f8549b-cdqdc\" (UID: \"e0a4cf62-6b8e-4e88-8c36-85597c9f2638\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d78a04d0-6bc8f8549b-cdqdc"
Jul 26 16:18:08.060153 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:08.060030    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dff03dcd-30c4-41ee-9cfb-83e9d3a8c4fb-plugins-config-volume\") pod \"payload-processing-e2e-migrate-d78a04d0-8479c4d7f7-j94px\" (UID: \"dff03dcd-30c4-41ee-9cfb-83e9d3a8c4fb\") " pod="openshift-ingress/payload-processing-e2e-migrate-d78a04d0-8479c4d7f7-j94px"
Jul 26 16:18:08.066823 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:08.066789    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktnjn\" (UniqueName: \"kubernetes.io/projected/dff03dcd-30c4-41ee-9cfb-83e9d3a8c4fb-kube-api-access-ktnjn\") pod \"payload-processing-e2e-migrate-d78a04d0-8479c4d7f7-j94px\" (UID: \"dff03dcd-30c4-41ee-9cfb-83e9d3a8c4fb\") " pod="openshift-ingress/payload-processing-e2e-migrate-d78a04d0-8479c4d7f7-j94px"
Jul 26 16:18:08.066823 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:08.066817    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxt48\" (UniqueName: \"kubernetes.io/projected/e0a4cf62-6b8e-4e88-8c36-85597c9f2638-kube-api-access-cxt48\") pod \"payload-pre-processing-e2e-migrate-d78a04d0-6bc8f8549b-cdqdc\" (UID: \"e0a4cf62-6b8e-4e88-8c36-85597c9f2638\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d78a04d0-6bc8f8549b-cdqdc"
Jul 26 16:18:08.102714 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:08.102680    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d78a04d0-6bc8f8549b-cdqdc"
Jul 26 16:18:08.156198 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:08.156154    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d78a04d0-8479c4d7f7-j94px"
Jul 26 16:18:08.263332 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:08.263287    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d78a04d0-6bc8f8549b-cdqdc"]
Jul 26 16:18:08.265456 ip-10-0-133-204 kubenswrapper[2563]: W0726 16:18:08.265416    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0a4cf62_6b8e_4e88_8c36_85597c9f2638.slice/crio-a39f10cea1f969e2dda9e32f844df85608268ef923e5b0d3215de778eeb8d729 WatchSource:0}: Error finding container a39f10cea1f969e2dda9e32f844df85608268ef923e5b0d3215de778eeb8d729: Status 404 returned error can't find the container with id a39f10cea1f969e2dda9e32f844df85608268ef923e5b0d3215de778eeb8d729
Jul 26 16:18:08.344371 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:08.344341    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d78a04d0-8479c4d7f7-j94px"]
Jul 26 16:18:08.346534 ip-10-0-133-204 kubenswrapper[2563]: W0726 16:18:08.346502    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddff03dcd_30c4_41ee_9cfb_83e9d3a8c4fb.slice/crio-feac690dc113fb9193e220fb13a9aa8e9e70ae1074b7689898ffeaa13d12310e WatchSource:0}: Error finding container feac690dc113fb9193e220fb13a9aa8e9e70ae1074b7689898ffeaa13d12310e: Status 404 returned error can't find the container with id feac690dc113fb9193e220fb13a9aa8e9e70ae1074b7689898ffeaa13d12310e
Jul 26 16:18:08.471832 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:08.471794    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d78a04d0-8479c4d7f7-j94px" event={"ID":"dff03dcd-30c4-41ee-9cfb-83e9d3a8c4fb","Type":"ContainerStarted","Data":"28d3296d351ab453c9ddd919a3303e5e91b3d22cc7e75b6af7e88861c4985bac"}
Jul 26 16:18:08.471983 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:08.471855    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d78a04d0-8479c4d7f7-j94px" event={"ID":"dff03dcd-30c4-41ee-9cfb-83e9d3a8c4fb","Type":"ContainerStarted","Data":"feac690dc113fb9193e220fb13a9aa8e9e70ae1074b7689898ffeaa13d12310e"}
Jul 26 16:18:08.471983 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:08.471896    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-d78a04d0-8479c4d7f7-j94px"
Jul 26 16:18:08.477016 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:08.476911    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d78a04d0-669d6d6fc4-6b8k9" event={"ID":"e17a5df2-c4f1-441d-af7a-6887f7db1c5e","Type":"ContainerStarted","Data":"8688e9c18405e67f0f7df3d2bfde019fbe31a42e367fc8a2e729998702ab11f0"}
Jul 26 16:18:08.477016 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:08.476961    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d78a04d0-669d6d6fc4-6b8k9" event={"ID":"e17a5df2-c4f1-441d-af7a-6887f7db1c5e","Type":"ContainerStarted","Data":"0cdcb866f23a9a0449ec42aea5cd2b04c40708d8eac1ec80822e1f8a6cc76a54"}
Jul 26 16:18:08.477016 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:08.477011    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d78a04d0-669d6d6fc4-6b8k9"
Jul 26 16:18:08.478727 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:08.478706    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d78a04d0-6bc8f8549b-cdqdc" event={"ID":"e0a4cf62-6b8e-4e88-8c36-85597c9f2638","Type":"ContainerStarted","Data":"212ee0e307cee7eaf93cc4c0d9056d13a6577e901f5d0ccfd46bdb12fdf0651c"}
Jul 26 16:18:08.478822 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:08.478746    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d78a04d0-6bc8f8549b-cdqdc" event={"ID":"e0a4cf62-6b8e-4e88-8c36-85597c9f2638","Type":"ContainerStarted","Data":"a39f10cea1f969e2dda9e32f844df85608268ef923e5b0d3215de778eeb8d729"}
Jul 26 16:18:08.478867 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:08.478815    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d78a04d0-6bc8f8549b-cdqdc"
Jul 26 16:18:08.491435 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:08.491386    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-d78a04d0-8479c4d7f7-j94px" podStartSLOduration=1.4913673219999999 podStartE2EDuration="1.491367322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:18:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:18:08.488176457 +0000 UTC m=+1680.592402285" watchObservedRunningTime="2026-07-26 16:18:08.491367322 +0000 UTC m=+1680.595593134"
Jul 26 16:18:08.505734 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:08.505663    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d78a04d0-669d6d6fc4-6b8k9" podStartSLOduration=2.223931435 podStartE2EDuration="2.505647664s" totalImagesPullingTime="281.716229ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:18:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:18:08.504688883 +0000 UTC m=+1680.608914692" watchObservedRunningTime="2026-07-26 16:18:08.505647664 +0000 UTC m=+1680.609873463"
Jul 26 16:18:08.519976 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:08.519916    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d78a04d0-6bc8f8549b-cdqdc" podStartSLOduration=1.51989282 podStartE2EDuration="1.51989282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:18:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:18:08.51925255 +0000 UTC m=+1680.623478380" watchObservedRunningTime="2026-07-26 16:18:08.51989282 +0000 UTC m=+1680.624118629"
Jul 26 16:18:14.488914 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:14.488883    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d78a04d0-669d6d6fc4-6b8k9"
Jul 26 16:18:19.485788 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:19.485743    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-d78a04d0-8479c4d7f7-j94px"
Jul 26 16:18:19.486215 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:19.485934    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d78a04d0-6bc8f8549b-cdqdc"
Jul 26 16:18:32.886513 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:32.886474    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:18:33.502690 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:33.502661    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:18:42.096332 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:42.096290    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d78a04d0-669d6d6fc4-6b8k9"]
Jul 26 16:18:42.096865 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:42.096544    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d78a04d0-669d6d6fc4-6b8k9" podUID="e17a5df2-c4f1-441d-af7a-6887f7db1c5e" containerName="maas-api" containerID="cri-o://8688e9c18405e67f0f7df3d2bfde019fbe31a42e367fc8a2e729998702ab11f0" gracePeriod=30
Jul 26 16:18:42.122572 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:42.122542    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:18:42.141213 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:42.141170    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d78a04d0-8479c4d7f7-j94px"]
Jul 26 16:18:42.141422 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:42.141395    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-d78a04d0-8479c4d7f7-j94px" podUID="dff03dcd-30c4-41ee-9cfb-83e9d3a8c4fb" containerName="payload-processing" containerID="cri-o://28d3296d351ab453c9ddd919a3303e5e91b3d22cc7e75b6af7e88861c4985bac" gracePeriod=30
Jul 26 16:18:42.150471 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:42.150443    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d78a04d0-6bc8f8549b-cdqdc"]
Jul 26 16:18:42.150657 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:42.150637    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d78a04d0-6bc8f8549b-cdqdc" podUID="e0a4cf62-6b8e-4e88-8c36-85597c9f2638" containerName="payload-pre-processing" containerID="cri-o://212ee0e307cee7eaf93cc4c0d9056d13a6577e901f5d0ccfd46bdb12fdf0651c" gracePeriod=30
Jul 26 16:18:42.186753 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:42.186721    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:18:42.288895 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:42.288861    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:18:42.416664 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:42.416640    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d78a04d0-669d6d6fc4-6b8k9"
Jul 26 16:18:42.453113 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:42.453091    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d78a04d0-8479c4d7f7-j94px"
Jul 26 16:18:42.458824 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:42.458800    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d78a04d0-6bc8f8549b-cdqdc"
Jul 26 16:18:42.584959 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:42.584924    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ktv8k\" (UniqueName: \"kubernetes.io/projected/e17a5df2-c4f1-441d-af7a-6887f7db1c5e-kube-api-access-ktv8k\") pod \"e17a5df2-c4f1-441d-af7a-6887f7db1c5e\" (UID: \"e17a5df2-c4f1-441d-af7a-6887f7db1c5e\") "
Jul 26 16:18:42.585179 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:42.585000    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cxt48\" (UniqueName: \"kubernetes.io/projected/e0a4cf62-6b8e-4e88-8c36-85597c9f2638-kube-api-access-cxt48\") pod \"e0a4cf62-6b8e-4e88-8c36-85597c9f2638\" (UID: \"e0a4cf62-6b8e-4e88-8c36-85597c9f2638\") "
Jul 26 16:18:42.585179 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:42.585084    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ktnjn\" (UniqueName: \"kubernetes.io/projected/dff03dcd-30c4-41ee-9cfb-83e9d3a8c4fb-kube-api-access-ktnjn\") pod \"dff03dcd-30c4-41ee-9cfb-83e9d3a8c4fb\" (UID: \"dff03dcd-30c4-41ee-9cfb-83e9d3a8c4fb\") "
Jul 26 16:18:42.585179 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:42.585132    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0a4cf62-6b8e-4e88-8c36-85597c9f2638-plugins-config-volume\") pod \"e0a4cf62-6b8e-4e88-8c36-85597c9f2638\" (UID: \"e0a4cf62-6b8e-4e88-8c36-85597c9f2638\") "
Jul 26 16:18:42.585179 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:42.585159    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e17a5df2-c4f1-441d-af7a-6887f7db1c5e-openshift-service-ca\") pod \"e17a5df2-c4f1-441d-af7a-6887f7db1c5e\" (UID: \"e17a5df2-c4f1-441d-af7a-6887f7db1c5e\") "
Jul 26 16:18:42.585733 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:42.585526    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0a4cf62-6b8e-4e88-8c36-85597c9f2638-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e0a4cf62-6b8e-4e88-8c36-85597c9f2638" (UID: "e0a4cf62-6b8e-4e88-8c36-85597c9f2638"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 26 16:18:42.585957 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:42.585741    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e17a5df2-c4f1-441d-af7a-6887f7db1c5e-maas-api-tls\") pod \"e17a5df2-c4f1-441d-af7a-6887f7db1c5e\" (UID: \"e17a5df2-c4f1-441d-af7a-6887f7db1c5e\") "
Jul 26 16:18:42.586107 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:42.586078    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e17a5df2-c4f1-441d-af7a-6887f7db1c5e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e17a5df2-c4f1-441d-af7a-6887f7db1c5e" (UID: "e17a5df2-c4f1-441d-af7a-6887f7db1c5e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 26 16:18:42.586470 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:42.586436    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dff03dcd-30c4-41ee-9cfb-83e9d3a8c4fb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "dff03dcd-30c4-41ee-9cfb-83e9d3a8c4fb" (UID: "dff03dcd-30c4-41ee-9cfb-83e9d3a8c4fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 26 16:18:42.587322 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:42.585797    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dff03dcd-30c4-41ee-9cfb-83e9d3a8c4fb-plugins-config-volume\") pod \"dff03dcd-30c4-41ee-9cfb-83e9d3a8c4fb\" (UID: \"dff03dcd-30c4-41ee-9cfb-83e9d3a8c4fb\") "
Jul 26 16:18:42.587668 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:42.587649    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0a4cf62-6b8e-4e88-8c36-85597c9f2638-plugins-config-volume\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:18:42.587756 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:42.587675    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e17a5df2-c4f1-441d-af7a-6887f7db1c5e-openshift-service-ca\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:18:42.587756 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:42.587690    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dff03dcd-30c4-41ee-9cfb-83e9d3a8c4fb-plugins-config-volume\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:18:42.587896 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:42.587870    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e17a5df2-c4f1-441d-af7a-6887f7db1c5e-kube-api-access-ktv8k" (OuterVolumeSpecName: "kube-api-access-ktv8k") pod "e17a5df2-c4f1-441d-af7a-6887f7db1c5e" (UID: "e17a5df2-c4f1-441d-af7a-6887f7db1c5e"). InnerVolumeSpecName "kube-api-access-ktv8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 26 16:18:42.587974 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:42.587953    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0a4cf62-6b8e-4e88-8c36-85597c9f2638-kube-api-access-cxt48" (OuterVolumeSpecName: "kube-api-access-cxt48") pod "e0a4cf62-6b8e-4e88-8c36-85597c9f2638" (UID: "e0a4cf62-6b8e-4e88-8c36-85597c9f2638"). InnerVolumeSpecName "kube-api-access-cxt48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 26 16:18:42.588087 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:42.588068    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dff03dcd-30c4-41ee-9cfb-83e9d3a8c4fb-kube-api-access-ktnjn" (OuterVolumeSpecName: "kube-api-access-ktnjn") pod "dff03dcd-30c4-41ee-9cfb-83e9d3a8c4fb" (UID: "dff03dcd-30c4-41ee-9cfb-83e9d3a8c4fb"). InnerVolumeSpecName "kube-api-access-ktnjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 26 16:18:42.588376 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:42.588341    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e17a5df2-c4f1-441d-af7a-6887f7db1c5e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e17a5df2-c4f1-441d-af7a-6887f7db1c5e" (UID: "e17a5df2-c4f1-441d-af7a-6887f7db1c5e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 26 16:18:42.640805 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:42.640687    2563 generic.go:356] "Generic (PLEG): container finished" podID="e0a4cf62-6b8e-4e88-8c36-85597c9f2638" containerID="212ee0e307cee7eaf93cc4c0d9056d13a6577e901f5d0ccfd46bdb12fdf0651c" exitCode=0
Jul 26 16:18:42.640805 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:42.640730    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d78a04d0-6bc8f8549b-cdqdc"
Jul 26 16:18:42.641058 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:42.640807    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d78a04d0-6bc8f8549b-cdqdc" event={"ID":"e0a4cf62-6b8e-4e88-8c36-85597c9f2638","Type":"ContainerDied","Data":"212ee0e307cee7eaf93cc4c0d9056d13a6577e901f5d0ccfd46bdb12fdf0651c"}
Jul 26 16:18:42.641058 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:42.640898    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d78a04d0-6bc8f8549b-cdqdc" event={"ID":"e0a4cf62-6b8e-4e88-8c36-85597c9f2638","Type":"ContainerDied","Data":"a39f10cea1f969e2dda9e32f844df85608268ef923e5b0d3215de778eeb8d729"}
Jul 26 16:18:42.641058 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:42.640931    2563 scope.go:117] "RemoveContainer" containerID="212ee0e307cee7eaf93cc4c0d9056d13a6577e901f5d0ccfd46bdb12fdf0651c"
Jul 26 16:18:42.642228 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:42.642210    2563 generic.go:356] "Generic (PLEG): container finished" podID="dff03dcd-30c4-41ee-9cfb-83e9d3a8c4fb" containerID="28d3296d351ab453c9ddd919a3303e5e91b3d22cc7e75b6af7e88861c4985bac" exitCode=0
Jul 26 16:18:42.642309 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:42.642258    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d78a04d0-8479c4d7f7-j94px"
Jul 26 16:18:42.642309 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:42.642294    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d78a04d0-8479c4d7f7-j94px" event={"ID":"dff03dcd-30c4-41ee-9cfb-83e9d3a8c4fb","Type":"ContainerDied","Data":"28d3296d351ab453c9ddd919a3303e5e91b3d22cc7e75b6af7e88861c4985bac"}
Jul 26 16:18:42.642424 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:42.642344    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d78a04d0-8479c4d7f7-j94px" event={"ID":"dff03dcd-30c4-41ee-9cfb-83e9d3a8c4fb","Type":"ContainerDied","Data":"feac690dc113fb9193e220fb13a9aa8e9e70ae1074b7689898ffeaa13d12310e"}
Jul 26 16:18:42.643539 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:42.643517    2563 generic.go:356] "Generic (PLEG): container finished" podID="e17a5df2-c4f1-441d-af7a-6887f7db1c5e" containerID="8688e9c18405e67f0f7df3d2bfde019fbe31a42e367fc8a2e729998702ab11f0" exitCode=0
Jul 26 16:18:42.643663 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:42.643555    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d78a04d0-669d6d6fc4-6b8k9" event={"ID":"e17a5df2-c4f1-441d-af7a-6887f7db1c5e","Type":"ContainerDied","Data":"8688e9c18405e67f0f7df3d2bfde019fbe31a42e367fc8a2e729998702ab11f0"}
Jul 26 16:18:42.643663 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:42.643571    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d78a04d0-669d6d6fc4-6b8k9"
Jul 26 16:18:42.643663 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:42.643591    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d78a04d0-669d6d6fc4-6b8k9" event={"ID":"e17a5df2-c4f1-441d-af7a-6887f7db1c5e","Type":"ContainerDied","Data":"0cdcb866f23a9a0449ec42aea5cd2b04c40708d8eac1ec80822e1f8a6cc76a54"}
Jul 26 16:18:42.650328 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:42.650305    2563 scope.go:117] "RemoveContainer" containerID="212ee0e307cee7eaf93cc4c0d9056d13a6577e901f5d0ccfd46bdb12fdf0651c"
Jul 26 16:18:42.650643 ip-10-0-133-204 kubenswrapper[2563]: E0726 16:18:42.650613    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"212ee0e307cee7eaf93cc4c0d9056d13a6577e901f5d0ccfd46bdb12fdf0651c\": container with ID starting with 212ee0e307cee7eaf93cc4c0d9056d13a6577e901f5d0ccfd46bdb12fdf0651c not found: ID does not exist" containerID="212ee0e307cee7eaf93cc4c0d9056d13a6577e901f5d0ccfd46bdb12fdf0651c"
Jul 26 16:18:42.650719 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:42.650655    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"212ee0e307cee7eaf93cc4c0d9056d13a6577e901f5d0ccfd46bdb12fdf0651c"} err="failed to get container status \"212ee0e307cee7eaf93cc4c0d9056d13a6577e901f5d0ccfd46bdb12fdf0651c\": rpc error: code = NotFound desc = could not find container \"212ee0e307cee7eaf93cc4c0d9056d13a6577e901f5d0ccfd46bdb12fdf0651c\": container with ID starting with 212ee0e307cee7eaf93cc4c0d9056d13a6577e901f5d0ccfd46bdb12fdf0651c not found: ID does not exist"
Jul 26 16:18:42.650719 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:42.650680    2563 scope.go:117] "RemoveContainer" containerID="28d3296d351ab453c9ddd919a3303e5e91b3d22cc7e75b6af7e88861c4985bac"
Jul 26 16:18:42.659803 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:42.659774    2563 scope.go:117] "RemoveContainer" containerID="28d3296d351ab453c9ddd919a3303e5e91b3d22cc7e75b6af7e88861c4985bac"
Jul 26 16:18:42.660128 ip-10-0-133-204 kubenswrapper[2563]: E0726 16:18:42.660108    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28d3296d351ab453c9ddd919a3303e5e91b3d22cc7e75b6af7e88861c4985bac\": container with ID starting with 28d3296d351ab453c9ddd919a3303e5e91b3d22cc7e75b6af7e88861c4985bac not found: ID does not exist" containerID="28d3296d351ab453c9ddd919a3303e5e91b3d22cc7e75b6af7e88861c4985bac"
Jul 26 16:18:42.660209 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:42.660135    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28d3296d351ab453c9ddd919a3303e5e91b3d22cc7e75b6af7e88861c4985bac"} err="failed to get container status \"28d3296d351ab453c9ddd919a3303e5e91b3d22cc7e75b6af7e88861c4985bac\": rpc error: code = NotFound desc = could not find container \"28d3296d351ab453c9ddd919a3303e5e91b3d22cc7e75b6af7e88861c4985bac\": container with ID starting with 28d3296d351ab453c9ddd919a3303e5e91b3d22cc7e75b6af7e88861c4985bac not found: ID does not exist"
Jul 26 16:18:42.660209 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:42.660153    2563 scope.go:117] "RemoveContainer" containerID="8688e9c18405e67f0f7df3d2bfde019fbe31a42e367fc8a2e729998702ab11f0"
Jul 26 16:18:42.668344 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:42.668323    2563 scope.go:117] "RemoveContainer" containerID="8688e9c18405e67f0f7df3d2bfde019fbe31a42e367fc8a2e729998702ab11f0"
Jul 26 16:18:42.668596 ip-10-0-133-204 kubenswrapper[2563]: E0726 16:18:42.668574    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8688e9c18405e67f0f7df3d2bfde019fbe31a42e367fc8a2e729998702ab11f0\": container with ID starting with 8688e9c18405e67f0f7df3d2bfde019fbe31a42e367fc8a2e729998702ab11f0 not found: ID does not exist" containerID="8688e9c18405e67f0f7df3d2bfde019fbe31a42e367fc8a2e729998702ab11f0"
Jul 26 16:18:42.668693 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:42.668603    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8688e9c18405e67f0f7df3d2bfde019fbe31a42e367fc8a2e729998702ab11f0"} err="failed to get container status \"8688e9c18405e67f0f7df3d2bfde019fbe31a42e367fc8a2e729998702ab11f0\": rpc error: code = NotFound desc = could not find container \"8688e9c18405e67f0f7df3d2bfde019fbe31a42e367fc8a2e729998702ab11f0\": container with ID starting with 8688e9c18405e67f0f7df3d2bfde019fbe31a42e367fc8a2e729998702ab11f0 not found: ID does not exist"
Jul 26 16:18:42.669908 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:42.669887    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d78a04d0-669d6d6fc4-6b8k9"]
Jul 26 16:18:42.673526 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:42.673488    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d78a04d0-669d6d6fc4-6b8k9"]
Jul 26 16:18:42.682913 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:42.682888    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d78a04d0-6bc8f8549b-cdqdc"]
Jul 26 16:18:42.687795 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:42.687756    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d78a04d0-6bc8f8549b-cdqdc"]
Jul 26 16:18:42.688284 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:42.688259    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktnjn\" (UniqueName: \"kubernetes.io/projected/dff03dcd-30c4-41ee-9cfb-83e9d3a8c4fb-kube-api-access-ktnjn\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:18:42.688384 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:42.688290    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e17a5df2-c4f1-441d-af7a-6887f7db1c5e-maas-api-tls\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:18:42.688384 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:42.688304    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktv8k\" (UniqueName: \"kubernetes.io/projected/e17a5df2-c4f1-441d-af7a-6887f7db1c5e-kube-api-access-ktv8k\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:18:42.688384 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:42.688317    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxt48\" (UniqueName: \"kubernetes.io/projected/e0a4cf62-6b8e-4e88-8c36-85597c9f2638-kube-api-access-cxt48\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:18:42.697124 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:42.697087    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d78a04d0-8479c4d7f7-j94px"]
Jul 26 16:18:42.698748 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:42.698726    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d78a04d0-8479c4d7f7-j94px"]
Jul 26 16:18:44.376830 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:44.375477    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dff03dcd-30c4-41ee-9cfb-83e9d3a8c4fb" path="/var/lib/kubelet/pods/dff03dcd-30c4-41ee-9cfb-83e9d3a8c4fb/volumes"
Jul 26 16:18:44.376830 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:44.376499    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0a4cf62-6b8e-4e88-8c36-85597c9f2638" path="/var/lib/kubelet/pods/e0a4cf62-6b8e-4e88-8c36-85597c9f2638/volumes"
Jul 26 16:18:44.377714 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:44.377692    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e17a5df2-c4f1-441d-af7a-6887f7db1c5e" path="/var/lib/kubelet/pods/e17a5df2-c4f1-441d-af7a-6887f7db1c5e/volumes"
Jul 26 16:18:46.782396 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:46.782345    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:18:46.934234 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:46.934193    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:18:47.209533 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:47.209425    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:18:47.403598 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:47.403566    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:18:47.724654 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:47.724624    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:18:48.258657 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:48.258600    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:18:49.789822 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:49.789782    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:18:49.876043 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:49.876009    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:18:49.876317 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:49.876297    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7a3c8582-58fbdcf88b-9jsws"]
Jul 26 16:18:49.876690 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:49.876678    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e17a5df2-c4f1-441d-af7a-6887f7db1c5e" containerName="maas-api"
Jul 26 16:18:49.876730 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:49.876692    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e17a5df2-c4f1-441d-af7a-6887f7db1c5e" containerName="maas-api"
Jul 26 16:18:49.876730 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:49.876713    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dff03dcd-30c4-41ee-9cfb-83e9d3a8c4fb" containerName="payload-processing"
Jul 26 16:18:49.876730 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:49.876719    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="dff03dcd-30c4-41ee-9cfb-83e9d3a8c4fb" containerName="payload-processing"
Jul 26 16:18:49.876868 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:49.876733    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0a4cf62-6b8e-4e88-8c36-85597c9f2638" containerName="payload-pre-processing"
Jul 26 16:18:49.876868 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:49.876738    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0a4cf62-6b8e-4e88-8c36-85597c9f2638" containerName="payload-pre-processing"
Jul 26 16:18:49.876868 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:49.876825    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e17a5df2-c4f1-441d-af7a-6887f7db1c5e" containerName="maas-api"
Jul 26 16:18:49.876868 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:49.876834    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0a4cf62-6b8e-4e88-8c36-85597c9f2638" containerName="payload-pre-processing"
Jul 26 16:18:49.876868 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:49.876840    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="dff03dcd-30c4-41ee-9cfb-83e9d3a8c4fb" containerName="payload-processing"
Jul 26 16:18:49.881632 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:49.881611    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7a3c8582-58fbdcf88b-9jsws"
Jul 26 16:18:49.884210 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:49.884187    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-7a3c8582-serving-cert\""
Jul 26 16:18:49.892929 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:49.892908    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7a3c8582-58fbdcf88b-9jsws"]
Jul 26 16:18:49.949420 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:49.949383    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:18:50.050881 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:50.050755    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:18:50.056805 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:50.056097    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k62bm\" (UniqueName: \"kubernetes.io/projected/4997b0b9-cc8f-4fdc-9659-09527ba3a3f4-kube-api-access-k62bm\") pod \"maas-api-e2e-ait-7a3c8582-58fbdcf88b-9jsws\" (UID: \"4997b0b9-cc8f-4fdc-9659-09527ba3a3f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7a3c8582-58fbdcf88b-9jsws"
Jul 26 16:18:50.056805 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:50.056157    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4997b0b9-cc8f-4fdc-9659-09527ba3a3f4-maas-api-tls\") pod \"maas-api-e2e-ait-7a3c8582-58fbdcf88b-9jsws\" (UID: \"4997b0b9-cc8f-4fdc-9659-09527ba3a3f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7a3c8582-58fbdcf88b-9jsws"
Jul 26 16:18:50.056805 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:50.056184    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4997b0b9-cc8f-4fdc-9659-09527ba3a3f4-openshift-service-ca\") pod \"maas-api-e2e-ait-7a3c8582-58fbdcf88b-9jsws\" (UID: \"4997b0b9-cc8f-4fdc-9659-09527ba3a3f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7a3c8582-58fbdcf88b-9jsws"
Jul 26 16:18:50.157174 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:50.157140    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4997b0b9-cc8f-4fdc-9659-09527ba3a3f4-maas-api-tls\") pod \"maas-api-e2e-ait-7a3c8582-58fbdcf88b-9jsws\" (UID: \"4997b0b9-cc8f-4fdc-9659-09527ba3a3f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7a3c8582-58fbdcf88b-9jsws"
Jul 26 16:18:50.157174 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:50.157175    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4997b0b9-cc8f-4fdc-9659-09527ba3a3f4-openshift-service-ca\") pod \"maas-api-e2e-ait-7a3c8582-58fbdcf88b-9jsws\" (UID: \"4997b0b9-cc8f-4fdc-9659-09527ba3a3f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7a3c8582-58fbdcf88b-9jsws"
Jul 26 16:18:50.157391 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:50.157267    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k62bm\" (UniqueName: \"kubernetes.io/projected/4997b0b9-cc8f-4fdc-9659-09527ba3a3f4-kube-api-access-k62bm\") pod \"maas-api-e2e-ait-7a3c8582-58fbdcf88b-9jsws\" (UID: \"4997b0b9-cc8f-4fdc-9659-09527ba3a3f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7a3c8582-58fbdcf88b-9jsws"
Jul 26 16:18:50.158005 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:50.157974    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4997b0b9-cc8f-4fdc-9659-09527ba3a3f4-openshift-service-ca\") pod \"maas-api-e2e-ait-7a3c8582-58fbdcf88b-9jsws\" (UID: \"4997b0b9-cc8f-4fdc-9659-09527ba3a3f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7a3c8582-58fbdcf88b-9jsws"
Jul 26 16:18:50.159658 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:50.159638    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4997b0b9-cc8f-4fdc-9659-09527ba3a3f4-maas-api-tls\") pod \"maas-api-e2e-ait-7a3c8582-58fbdcf88b-9jsws\" (UID: \"4997b0b9-cc8f-4fdc-9659-09527ba3a3f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7a3c8582-58fbdcf88b-9jsws"
Jul 26 16:18:50.165749 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:50.165726    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k62bm\" (UniqueName: \"kubernetes.io/projected/4997b0b9-cc8f-4fdc-9659-09527ba3a3f4-kube-api-access-k62bm\") pod \"maas-api-e2e-ait-7a3c8582-58fbdcf88b-9jsws\" (UID: \"4997b0b9-cc8f-4fdc-9659-09527ba3a3f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7a3c8582-58fbdcf88b-9jsws"
Jul 26 16:18:50.197416 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:50.197378    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7a3c8582-58fbdcf88b-9jsws"
Jul 26 16:18:50.358754 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:50.358731    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7a3c8582-58fbdcf88b-9jsws"]
Jul 26 16:18:50.360971 ip-10-0-133-204 kubenswrapper[2563]: W0726 16:18:50.360941    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4997b0b9_cc8f_4fdc_9659_09527ba3a3f4.slice/crio-4e98101d48b051ccaedbb89bb1c0ff14aa4afd9acc8f6357d852fab371920b13 WatchSource:0}: Error finding container 4e98101d48b051ccaedbb89bb1c0ff14aa4afd9acc8f6357d852fab371920b13: Status 404 returned error can't find the container with id 4e98101d48b051ccaedbb89bb1c0ff14aa4afd9acc8f6357d852fab371920b13
Jul 26 16:18:50.376437 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:50.376412    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:18:50.473757 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:50.473715    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7a3c8582-cf5f7c5b4-5vmj9"]
Jul 26 16:18:50.474448 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:50.474419    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:18:50.478598 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:50.478581    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7a3c8582-cf5f7c5b4-5vmj9"
Jul 26 16:18:50.480976 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:50.480957    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-7a3c8582-dockercfg-njbcw\""
Jul 26 16:18:50.481156 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:50.481140    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-7a3c8582\""
Jul 26 16:18:50.491807 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:50.486010    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7a3c8582-cf5f7c5b4-5vmj9"]
Jul 26 16:18:50.511537 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:50.511497    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7a3c8582-56cbfdcc8-7qgkp"]
Jul 26 16:18:50.515116 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:50.515094    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7a3c8582-56cbfdcc8-7qgkp"
Jul 26 16:18:50.524315 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:50.524291    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7a3c8582-56cbfdcc8-7qgkp"]
Jul 26 16:18:50.662087 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:50.662003    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qch9g\" (UniqueName: \"kubernetes.io/projected/5730c475-a7bd-4a00-8b4b-e8feb3db590b-kube-api-access-qch9g\") pod \"payload-pre-processing-e2e-ait-7a3c8582-cf5f7c5b4-5vmj9\" (UID: \"5730c475-a7bd-4a00-8b4b-e8feb3db590b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7a3c8582-cf5f7c5b4-5vmj9"
Jul 26 16:18:50.662087 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:50.662057    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h98jh\" (UniqueName: \"kubernetes.io/projected/a922ec4d-a138-426f-8caa-4735c6cb71b8-kube-api-access-h98jh\") pod \"payload-processing-e2e-ait-7a3c8582-56cbfdcc8-7qgkp\" (UID: \"a922ec4d-a138-426f-8caa-4735c6cb71b8\") " pod="openshift-ingress/payload-processing-e2e-ait-7a3c8582-56cbfdcc8-7qgkp"
Jul 26 16:18:50.662280 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:50.662170    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a922ec4d-a138-426f-8caa-4735c6cb71b8-plugins-config-volume\") pod \"payload-processing-e2e-ait-7a3c8582-56cbfdcc8-7qgkp\" (UID: \"a922ec4d-a138-426f-8caa-4735c6cb71b8\") " pod="openshift-ingress/payload-processing-e2e-ait-7a3c8582-56cbfdcc8-7qgkp"
Jul 26 16:18:50.662280 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:50.662218    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5730c475-a7bd-4a00-8b4b-e8feb3db590b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7a3c8582-cf5f7c5b4-5vmj9\" (UID: \"5730c475-a7bd-4a00-8b4b-e8feb3db590b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7a3c8582-cf5f7c5b4-5vmj9"
Jul 26 16:18:50.687108 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:50.687067    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7a3c8582-58fbdcf88b-9jsws" event={"ID":"4997b0b9-cc8f-4fdc-9659-09527ba3a3f4","Type":"ContainerStarted","Data":"4e98101d48b051ccaedbb89bb1c0ff14aa4afd9acc8f6357d852fab371920b13"}
Jul 26 16:18:50.762788 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:50.762729    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a922ec4d-a138-426f-8caa-4735c6cb71b8-plugins-config-volume\") pod \"payload-processing-e2e-ait-7a3c8582-56cbfdcc8-7qgkp\" (UID: \"a922ec4d-a138-426f-8caa-4735c6cb71b8\") " pod="openshift-ingress/payload-processing-e2e-ait-7a3c8582-56cbfdcc8-7qgkp"
Jul 26 16:18:50.762971 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:50.762811    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5730c475-a7bd-4a00-8b4b-e8feb3db590b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7a3c8582-cf5f7c5b4-5vmj9\" (UID: \"5730c475-a7bd-4a00-8b4b-e8feb3db590b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7a3c8582-cf5f7c5b4-5vmj9"
Jul 26 16:18:50.762971 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:50.762846    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qch9g\" (UniqueName: \"kubernetes.io/projected/5730c475-a7bd-4a00-8b4b-e8feb3db590b-kube-api-access-qch9g\") pod \"payload-pre-processing-e2e-ait-7a3c8582-cf5f7c5b4-5vmj9\" (UID: \"5730c475-a7bd-4a00-8b4b-e8feb3db590b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7a3c8582-cf5f7c5b4-5vmj9"
Jul 26 16:18:50.762971 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:50.762877    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h98jh\" (UniqueName: \"kubernetes.io/projected/a922ec4d-a138-426f-8caa-4735c6cb71b8-kube-api-access-h98jh\") pod \"payload-processing-e2e-ait-7a3c8582-56cbfdcc8-7qgkp\" (UID: \"a922ec4d-a138-426f-8caa-4735c6cb71b8\") " pod="openshift-ingress/payload-processing-e2e-ait-7a3c8582-56cbfdcc8-7qgkp"
Jul 26 16:18:50.763706 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:50.763679    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a922ec4d-a138-426f-8caa-4735c6cb71b8-plugins-config-volume\") pod \"payload-processing-e2e-ait-7a3c8582-56cbfdcc8-7qgkp\" (UID: \"a922ec4d-a138-426f-8caa-4735c6cb71b8\") " pod="openshift-ingress/payload-processing-e2e-ait-7a3c8582-56cbfdcc8-7qgkp"
Jul 26 16:18:50.763747 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:50.763679    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5730c475-a7bd-4a00-8b4b-e8feb3db590b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7a3c8582-cf5f7c5b4-5vmj9\" (UID: \"5730c475-a7bd-4a00-8b4b-e8feb3db590b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7a3c8582-cf5f7c5b4-5vmj9"
Jul 26 16:18:50.770874 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:50.770850    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h98jh\" (UniqueName: \"kubernetes.io/projected/a922ec4d-a138-426f-8caa-4735c6cb71b8-kube-api-access-h98jh\") pod \"payload-processing-e2e-ait-7a3c8582-56cbfdcc8-7qgkp\" (UID: \"a922ec4d-a138-426f-8caa-4735c6cb71b8\") " pod="openshift-ingress/payload-processing-e2e-ait-7a3c8582-56cbfdcc8-7qgkp"
Jul 26 16:18:50.770991 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:50.770920    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qch9g\" (UniqueName: \"kubernetes.io/projected/5730c475-a7bd-4a00-8b4b-e8feb3db590b-kube-api-access-qch9g\") pod \"payload-pre-processing-e2e-ait-7a3c8582-cf5f7c5b4-5vmj9\" (UID: \"5730c475-a7bd-4a00-8b4b-e8feb3db590b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7a3c8582-cf5f7c5b4-5vmj9"
Jul 26 16:18:50.793940 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:50.793906    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7a3c8582-cf5f7c5b4-5vmj9"
Jul 26 16:18:50.826342 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:50.826233    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7a3c8582-56cbfdcc8-7qgkp"
Jul 26 16:18:50.954556 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:50.954396    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7a3c8582-cf5f7c5b4-5vmj9"]
Jul 26 16:18:50.958275 ip-10-0-133-204 kubenswrapper[2563]: W0726 16:18:50.958236    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5730c475_a7bd_4a00_8b4b_e8feb3db590b.slice/crio-cc1a3434be7757f030b4f5ba7430b18a5e041142800e087b1bad100e7e14e89a WatchSource:0}: Error finding container cc1a3434be7757f030b4f5ba7430b18a5e041142800e087b1bad100e7e14e89a: Status 404 returned error can't find the container with id cc1a3434be7757f030b4f5ba7430b18a5e041142800e087b1bad100e7e14e89a
Jul 26 16:18:50.994106 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:50.993642    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7a3c8582-56cbfdcc8-7qgkp"]
Jul 26 16:18:51.694706 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:51.694670    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7a3c8582-cf5f7c5b4-5vmj9" event={"ID":"5730c475-a7bd-4a00-8b4b-e8feb3db590b","Type":"ContainerStarted","Data":"5adfcd6f06870e1a0bf16e899713b76a034431d3e9e3ce065e8ef81105ed4625"}
Jul 26 16:18:51.694953 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:51.694731    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7a3c8582-cf5f7c5b4-5vmj9" event={"ID":"5730c475-a7bd-4a00-8b4b-e8feb3db590b","Type":"ContainerStarted","Data":"cc1a3434be7757f030b4f5ba7430b18a5e041142800e087b1bad100e7e14e89a"}
Jul 26 16:18:51.694953 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:51.694915    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-7a3c8582-cf5f7c5b4-5vmj9"
Jul 26 16:18:51.696487 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:51.696459    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7a3c8582-58fbdcf88b-9jsws" event={"ID":"4997b0b9-cc8f-4fdc-9659-09527ba3a3f4","Type":"ContainerStarted","Data":"14a8a366e5e1634a2830e6909971e738723dc0aa4e173aa4337183951fdc9feb"}
Jul 26 16:18:51.696618 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:51.696510    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7a3c8582-58fbdcf88b-9jsws"
Jul 26 16:18:51.698180 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:51.698157    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7a3c8582-56cbfdcc8-7qgkp" event={"ID":"a922ec4d-a138-426f-8caa-4735c6cb71b8","Type":"ContainerStarted","Data":"0fa484b00221cf7ce2770de778c2478cc7c02c3d7a68f30b73c18bbca53decde"}
Jul 26 16:18:51.698271 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:51.698197    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7a3c8582-56cbfdcc8-7qgkp" event={"ID":"a922ec4d-a138-426f-8caa-4735c6cb71b8","Type":"ContainerStarted","Data":"0f8e071a55777a4ba58052b40620359a22800db933feebba9196d49a3c604662"}
Jul 26 16:18:51.698271 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:51.698226    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-7a3c8582-56cbfdcc8-7qgkp"
Jul 26 16:18:51.709410 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:51.709357    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-7a3c8582-cf5f7c5b4-5vmj9" podStartSLOduration=1.709336838 podStartE2EDuration="1.709336838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:18:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:18:51.708620224 +0000 UTC m=+1723.812846054" watchObservedRunningTime="2026-07-26 16:18:51.709336838 +0000 UTC m=+1723.813562644"
Jul 26 16:18:51.723903 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:51.723851    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-7a3c8582-56cbfdcc8-7qgkp" podStartSLOduration=1.723834456 podStartE2EDuration="1.723834456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:18:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:18:51.722460896 +0000 UTC m=+1723.826686723" watchObservedRunningTime="2026-07-26 16:18:51.723834456 +0000 UTC m=+1723.828060265"
Jul 26 16:18:51.737982 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:51.737928    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7a3c8582-58fbdcf88b-9jsws" podStartSLOduration=2.262785675 podStartE2EDuration="2.737911848s" totalImagesPullingTime="475.126173ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:18:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:18:51.736443461 +0000 UTC m=+1723.840669271" watchObservedRunningTime="2026-07-26 16:18:51.737911848 +0000 UTC m=+1723.842137658"
Jul 26 16:18:57.708910 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:18:57.708881    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7a3c8582-58fbdcf88b-9jsws"
Jul 26 16:19:02.705965 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:02.705936    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-7a3c8582-56cbfdcc8-7qgkp"
Jul 26 16:19:02.706553 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:02.706021    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-7a3c8582-cf5f7c5b4-5vmj9"
Jul 26 16:19:26.015074 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:26.014990    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7a3c8582-58fbdcf88b-9jsws"]
Jul 26 16:19:26.015606 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:26.015263    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7a3c8582-58fbdcf88b-9jsws" podUID="4997b0b9-cc8f-4fdc-9659-09527ba3a3f4" containerName="maas-api" containerID="cri-o://14a8a366e5e1634a2830e6909971e738723dc0aa4e173aa4337183951fdc9feb" gracePeriod=30
Jul 26 16:19:26.044463 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:26.044434    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:19:26.071998 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:26.071959    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7a3c8582-56cbfdcc8-7qgkp"]
Jul 26 16:19:26.072198 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:26.072169    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-7a3c8582-56cbfdcc8-7qgkp" podUID="a922ec4d-a138-426f-8caa-4735c6cb71b8" containerName="payload-processing" containerID="cri-o://0fa484b00221cf7ce2770de778c2478cc7c02c3d7a68f30b73c18bbca53decde" gracePeriod=30
Jul 26 16:19:26.081325 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:26.080976    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7a3c8582-cf5f7c5b4-5vmj9"]
Jul 26 16:19:26.081786 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:26.081733    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-7a3c8582-cf5f7c5b4-5vmj9" podUID="5730c475-a7bd-4a00-8b4b-e8feb3db590b" containerName="payload-pre-processing" containerID="cri-o://5adfcd6f06870e1a0bf16e899713b76a034431d3e9e3ce065e8ef81105ed4625" gracePeriod=30
Jul 26 16:19:26.112865 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:26.112832    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:19:26.169026 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:26.168995    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:19:26.367072 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:26.367046    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7a3c8582-56cbfdcc8-7qgkp"
Jul 26 16:19:26.416577 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:26.416549    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7a3c8582-cf5f7c5b4-5vmj9"
Jul 26 16:19:26.422342 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:26.422316    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7a3c8582-58fbdcf88b-9jsws"
Jul 26 16:19:26.489438 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:26.489401    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h98jh\" (UniqueName: \"kubernetes.io/projected/a922ec4d-a138-426f-8caa-4735c6cb71b8-kube-api-access-h98jh\") pod \"a922ec4d-a138-426f-8caa-4735c6cb71b8\" (UID: \"a922ec4d-a138-426f-8caa-4735c6cb71b8\") "
Jul 26 16:19:26.489618 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:26.489599    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a922ec4d-a138-426f-8caa-4735c6cb71b8-plugins-config-volume\") pod \"a922ec4d-a138-426f-8caa-4735c6cb71b8\" (UID: \"a922ec4d-a138-426f-8caa-4735c6cb71b8\") "
Jul 26 16:19:26.490066 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:26.490035    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a922ec4d-a138-426f-8caa-4735c6cb71b8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a922ec4d-a138-426f-8caa-4735c6cb71b8" (UID: "a922ec4d-a138-426f-8caa-4735c6cb71b8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 26 16:19:26.491627 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:26.491602    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a922ec4d-a138-426f-8caa-4735c6cb71b8-kube-api-access-h98jh" (OuterVolumeSpecName: "kube-api-access-h98jh") pod "a922ec4d-a138-426f-8caa-4735c6cb71b8" (UID: "a922ec4d-a138-426f-8caa-4735c6cb71b8"). InnerVolumeSpecName "kube-api-access-h98jh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 26 16:19:26.591376 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:26.591291    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k62bm\" (UniqueName: \"kubernetes.io/projected/4997b0b9-cc8f-4fdc-9659-09527ba3a3f4-kube-api-access-k62bm\") pod \"4997b0b9-cc8f-4fdc-9659-09527ba3a3f4\" (UID: \"4997b0b9-cc8f-4fdc-9659-09527ba3a3f4\") "
Jul 26 16:19:26.591376 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:26.591354    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4997b0b9-cc8f-4fdc-9659-09527ba3a3f4-maas-api-tls\") pod \"4997b0b9-cc8f-4fdc-9659-09527ba3a3f4\" (UID: \"4997b0b9-cc8f-4fdc-9659-09527ba3a3f4\") "
Jul 26 16:19:26.591581 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:26.591534    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4997b0b9-cc8f-4fdc-9659-09527ba3a3f4-openshift-service-ca\") pod \"4997b0b9-cc8f-4fdc-9659-09527ba3a3f4\" (UID: \"4997b0b9-cc8f-4fdc-9659-09527ba3a3f4\") "
Jul 26 16:19:26.591636 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:26.591596    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qch9g\" (UniqueName: \"kubernetes.io/projected/5730c475-a7bd-4a00-8b4b-e8feb3db590b-kube-api-access-qch9g\") pod \"5730c475-a7bd-4a00-8b4b-e8feb3db590b\" (UID: \"5730c475-a7bd-4a00-8b4b-e8feb3db590b\") "
Jul 26 16:19:26.591636 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:26.591624    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5730c475-a7bd-4a00-8b4b-e8feb3db590b-plugins-config-volume\") pod \"5730c475-a7bd-4a00-8b4b-e8feb3db590b\" (UID: \"5730c475-a7bd-4a00-8b4b-e8feb3db590b\") "
Jul 26 16:19:26.591857 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:26.591818    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4997b0b9-cc8f-4fdc-9659-09527ba3a3f4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4997b0b9-cc8f-4fdc-9659-09527ba3a3f4" (UID: "4997b0b9-cc8f-4fdc-9659-09527ba3a3f4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 26 16:19:26.592076 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:26.592058    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4997b0b9-cc8f-4fdc-9659-09527ba3a3f4-openshift-service-ca\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:19:26.592142 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:26.592083    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a922ec4d-a138-426f-8caa-4735c6cb71b8-plugins-config-volume\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:19:26.592142 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:26.592099    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h98jh\" (UniqueName: \"kubernetes.io/projected/a922ec4d-a138-426f-8caa-4735c6cb71b8-kube-api-access-h98jh\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:19:26.592238 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:26.592174    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5730c475-a7bd-4a00-8b4b-e8feb3db590b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5730c475-a7bd-4a00-8b4b-e8feb3db590b" (UID: "5730c475-a7bd-4a00-8b4b-e8feb3db590b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 26 16:19:26.593655 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:26.593633    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5730c475-a7bd-4a00-8b4b-e8feb3db590b-kube-api-access-qch9g" (OuterVolumeSpecName: "kube-api-access-qch9g") pod "5730c475-a7bd-4a00-8b4b-e8feb3db590b" (UID: "5730c475-a7bd-4a00-8b4b-e8feb3db590b"). InnerVolumeSpecName "kube-api-access-qch9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 26 16:19:26.593869 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:26.593846    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4997b0b9-cc8f-4fdc-9659-09527ba3a3f4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4997b0b9-cc8f-4fdc-9659-09527ba3a3f4" (UID: "4997b0b9-cc8f-4fdc-9659-09527ba3a3f4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 26 16:19:26.593951 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:26.593892    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4997b0b9-cc8f-4fdc-9659-09527ba3a3f4-kube-api-access-k62bm" (OuterVolumeSpecName: "kube-api-access-k62bm") pod "4997b0b9-cc8f-4fdc-9659-09527ba3a3f4" (UID: "4997b0b9-cc8f-4fdc-9659-09527ba3a3f4"). InnerVolumeSpecName "kube-api-access-k62bm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 26 16:19:26.693388 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:26.693351    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k62bm\" (UniqueName: \"kubernetes.io/projected/4997b0b9-cc8f-4fdc-9659-09527ba3a3f4-kube-api-access-k62bm\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:19:26.693388 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:26.693380    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4997b0b9-cc8f-4fdc-9659-09527ba3a3f4-maas-api-tls\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:19:26.693388 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:26.693390    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qch9g\" (UniqueName: \"kubernetes.io/projected/5730c475-a7bd-4a00-8b4b-e8feb3db590b-kube-api-access-qch9g\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:19:26.693619 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:26.693401    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5730c475-a7bd-4a00-8b4b-e8feb3db590b-plugins-config-volume\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:19:26.875375 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:26.875273    2563 generic.go:356] "Generic (PLEG): container finished" podID="5730c475-a7bd-4a00-8b4b-e8feb3db590b" containerID="5adfcd6f06870e1a0bf16e899713b76a034431d3e9e3ce065e8ef81105ed4625" exitCode=0
Jul 26 16:19:26.875375 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:26.875332    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7a3c8582-cf5f7c5b4-5vmj9"
Jul 26 16:19:26.875375 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:26.875356    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7a3c8582-cf5f7c5b4-5vmj9" event={"ID":"5730c475-a7bd-4a00-8b4b-e8feb3db590b","Type":"ContainerDied","Data":"5adfcd6f06870e1a0bf16e899713b76a034431d3e9e3ce065e8ef81105ed4625"}
Jul 26 16:19:26.875679 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:26.875447    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7a3c8582-cf5f7c5b4-5vmj9" event={"ID":"5730c475-a7bd-4a00-8b4b-e8feb3db590b","Type":"ContainerDied","Data":"cc1a3434be7757f030b4f5ba7430b18a5e041142800e087b1bad100e7e14e89a"}
Jul 26 16:19:26.875679 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:26.875476    2563 scope.go:117] "RemoveContainer" containerID="5adfcd6f06870e1a0bf16e899713b76a034431d3e9e3ce065e8ef81105ed4625"
Jul 26 16:19:26.876698 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:26.876678    2563 generic.go:356] "Generic (PLEG): container finished" podID="4997b0b9-cc8f-4fdc-9659-09527ba3a3f4" containerID="14a8a366e5e1634a2830e6909971e738723dc0aa4e173aa4337183951fdc9feb" exitCode=0
Jul 26 16:19:26.876818 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:26.876739    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7a3c8582-58fbdcf88b-9jsws"
Jul 26 16:19:26.876818 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:26.876748    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7a3c8582-58fbdcf88b-9jsws" event={"ID":"4997b0b9-cc8f-4fdc-9659-09527ba3a3f4","Type":"ContainerDied","Data":"14a8a366e5e1634a2830e6909971e738723dc0aa4e173aa4337183951fdc9feb"}
Jul 26 16:19:26.876818 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:26.876809    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7a3c8582-58fbdcf88b-9jsws" event={"ID":"4997b0b9-cc8f-4fdc-9659-09527ba3a3f4","Type":"ContainerDied","Data":"4e98101d48b051ccaedbb89bb1c0ff14aa4afd9acc8f6357d852fab371920b13"}
Jul 26 16:19:26.878349 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:26.878321    2563 generic.go:356] "Generic (PLEG): container finished" podID="a922ec4d-a138-426f-8caa-4735c6cb71b8" containerID="0fa484b00221cf7ce2770de778c2478cc7c02c3d7a68f30b73c18bbca53decde" exitCode=0
Jul 26 16:19:26.878465 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:26.878385    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7a3c8582-56cbfdcc8-7qgkp"
Jul 26 16:19:26.878465 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:26.878403    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7a3c8582-56cbfdcc8-7qgkp" event={"ID":"a922ec4d-a138-426f-8caa-4735c6cb71b8","Type":"ContainerDied","Data":"0fa484b00221cf7ce2770de778c2478cc7c02c3d7a68f30b73c18bbca53decde"}
Jul 26 16:19:26.878563 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:26.878466    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7a3c8582-56cbfdcc8-7qgkp" event={"ID":"a922ec4d-a138-426f-8caa-4735c6cb71b8","Type":"ContainerDied","Data":"0f8e071a55777a4ba58052b40620359a22800db933feebba9196d49a3c604662"}
Jul 26 16:19:26.884735 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:26.884716    2563 scope.go:117] "RemoveContainer" containerID="5adfcd6f06870e1a0bf16e899713b76a034431d3e9e3ce065e8ef81105ed4625"
Jul 26 16:19:26.885149 ip-10-0-133-204 kubenswrapper[2563]: E0726 16:19:26.885113    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5adfcd6f06870e1a0bf16e899713b76a034431d3e9e3ce065e8ef81105ed4625\": container with ID starting with 5adfcd6f06870e1a0bf16e899713b76a034431d3e9e3ce065e8ef81105ed4625 not found: ID does not exist" containerID="5adfcd6f06870e1a0bf16e899713b76a034431d3e9e3ce065e8ef81105ed4625"
Jul 26 16:19:26.885255 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:26.885150    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5adfcd6f06870e1a0bf16e899713b76a034431d3e9e3ce065e8ef81105ed4625"} err="failed to get container status \"5adfcd6f06870e1a0bf16e899713b76a034431d3e9e3ce065e8ef81105ed4625\": rpc error: code = NotFound desc = could not find container \"5adfcd6f06870e1a0bf16e899713b76a034431d3e9e3ce065e8ef81105ed4625\": container with ID starting with 5adfcd6f06870e1a0bf16e899713b76a034431d3e9e3ce065e8ef81105ed4625 not found: ID does not exist"
Jul 26 16:19:26.885255 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:26.885175    2563 scope.go:117] "RemoveContainer" containerID="14a8a366e5e1634a2830e6909971e738723dc0aa4e173aa4337183951fdc9feb"
Jul 26 16:19:26.895754 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:26.895734    2563 scope.go:117] "RemoveContainer" containerID="14a8a366e5e1634a2830e6909971e738723dc0aa4e173aa4337183951fdc9feb"
Jul 26 16:19:26.896036 ip-10-0-133-204 kubenswrapper[2563]: E0726 16:19:26.896017    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14a8a366e5e1634a2830e6909971e738723dc0aa4e173aa4337183951fdc9feb\": container with ID starting with 14a8a366e5e1634a2830e6909971e738723dc0aa4e173aa4337183951fdc9feb not found: ID does not exist" containerID="14a8a366e5e1634a2830e6909971e738723dc0aa4e173aa4337183951fdc9feb"
Jul 26 16:19:26.896107 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:26.896041    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14a8a366e5e1634a2830e6909971e738723dc0aa4e173aa4337183951fdc9feb"} err="failed to get container status \"14a8a366e5e1634a2830e6909971e738723dc0aa4e173aa4337183951fdc9feb\": rpc error: code = NotFound desc = could not find container \"14a8a366e5e1634a2830e6909971e738723dc0aa4e173aa4337183951fdc9feb\": container with ID starting with 14a8a366e5e1634a2830e6909971e738723dc0aa4e173aa4337183951fdc9feb not found: ID does not exist"
Jul 26 16:19:26.896107 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:26.896059    2563 scope.go:117] "RemoveContainer" containerID="0fa484b00221cf7ce2770de778c2478cc7c02c3d7a68f30b73c18bbca53decde"
Jul 26 16:19:26.906316 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:26.906289    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7a3c8582-cf5f7c5b4-5vmj9"]
Jul 26 16:19:26.907951 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:26.907934    2563 scope.go:117] "RemoveContainer" containerID="0fa484b00221cf7ce2770de778c2478cc7c02c3d7a68f30b73c18bbca53decde"
Jul 26 16:19:26.908262 ip-10-0-133-204 kubenswrapper[2563]: E0726 16:19:26.908244    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fa484b00221cf7ce2770de778c2478cc7c02c3d7a68f30b73c18bbca53decde\": container with ID starting with 0fa484b00221cf7ce2770de778c2478cc7c02c3d7a68f30b73c18bbca53decde not found: ID does not exist" containerID="0fa484b00221cf7ce2770de778c2478cc7c02c3d7a68f30b73c18bbca53decde"
Jul 26 16:19:26.908341 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:26.908269    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fa484b00221cf7ce2770de778c2478cc7c02c3d7a68f30b73c18bbca53decde"} err="failed to get container status \"0fa484b00221cf7ce2770de778c2478cc7c02c3d7a68f30b73c18bbca53decde\": rpc error: code = NotFound desc = could not find container \"0fa484b00221cf7ce2770de778c2478cc7c02c3d7a68f30b73c18bbca53decde\": container with ID starting with 0fa484b00221cf7ce2770de778c2478cc7c02c3d7a68f30b73c18bbca53decde not found: ID does not exist"
Jul 26 16:19:26.909313 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:26.909291    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7a3c8582-cf5f7c5b4-5vmj9"]
Jul 26 16:19:26.921446 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:26.921420    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7a3c8582-58fbdcf88b-9jsws"]
Jul 26 16:19:26.924948 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:26.924917    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7a3c8582-58fbdcf88b-9jsws"]
Jul 26 16:19:26.934427 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:26.934402    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7a3c8582-56cbfdcc8-7qgkp"]
Jul 26 16:19:26.940324 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:26.940296    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7a3c8582-56cbfdcc8-7qgkp"]
Jul 26 16:19:28.373219 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:28.373185    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4997b0b9-cc8f-4fdc-9659-09527ba3a3f4" path="/var/lib/kubelet/pods/4997b0b9-cc8f-4fdc-9659-09527ba3a3f4/volumes"
Jul 26 16:19:28.373582 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:28.373557    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5730c475-a7bd-4a00-8b4b-e8feb3db590b" path="/var/lib/kubelet/pods/5730c475-a7bd-4a00-8b4b-e8feb3db590b/volumes"
Jul 26 16:19:28.373884 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:28.373871    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a922ec4d-a138-426f-8caa-4735c6cb71b8" path="/var/lib/kubelet/pods/a922ec4d-a138-426f-8caa-4735c6cb71b8/volumes"
Jul 26 16:19:30.701288 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:30.701243    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:19:31.881976 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:31.881946    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:19:32.108554 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:32.108473    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:19:32.329433 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:32.329396    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:19:32.638215 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:32.638112    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:19:32.699071 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:32.699030    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:19:35.156066 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:35.156020    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:19:35.240469 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:35.240430    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:19:35.246502 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:35.246421    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f22ea10f-5b5f999f7f-jq52n"]
Jul 26 16:19:35.247051 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:35.247026    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5730c475-a7bd-4a00-8b4b-e8feb3db590b" containerName="payload-pre-processing"
Jul 26 16:19:35.247051 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:35.247047    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5730c475-a7bd-4a00-8b4b-e8feb3db590b" containerName="payload-pre-processing"
Jul 26 16:19:35.247256 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:35.247063    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a922ec4d-a138-426f-8caa-4735c6cb71b8" containerName="payload-processing"
Jul 26 16:19:35.247256 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:35.247072    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a922ec4d-a138-426f-8caa-4735c6cb71b8" containerName="payload-processing"
Jul 26 16:19:35.247256 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:35.247099    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4997b0b9-cc8f-4fdc-9659-09527ba3a3f4" containerName="maas-api"
Jul 26 16:19:35.247256 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:35.247108    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4997b0b9-cc8f-4fdc-9659-09527ba3a3f4" containerName="maas-api"
Jul 26 16:19:35.247256 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:35.247206    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a922ec4d-a138-426f-8caa-4735c6cb71b8" containerName="payload-processing"
Jul 26 16:19:35.247256 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:35.247221    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4997b0b9-cc8f-4fdc-9659-09527ba3a3f4" containerName="maas-api"
Jul 26 16:19:35.247256 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:35.247232    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5730c475-a7bd-4a00-8b4b-e8feb3db590b" containerName="payload-pre-processing"
Jul 26 16:19:35.251807 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:35.251755    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f22ea10f-5b5f999f7f-jq52n"
Jul 26 16:19:35.254369 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:35.254339    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-f22ea10f-serving-cert\""
Jul 26 16:19:35.261073 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:35.261038    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f22ea10f-5b5f999f7f-jq52n"]
Jul 26 16:19:35.272920 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:35.272894    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb60c02f-e91d-4b98-a61c-2bfe455bbe31-maas-api-tls\") pod \"maas-api-e2e-derive-f22ea10f-5b5f999f7f-jq52n\" (UID: \"eb60c02f-e91d-4b98-a61c-2bfe455bbe31\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f22ea10f-5b5f999f7f-jq52n"
Jul 26 16:19:35.273052 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:35.272925    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbpl5\" (UniqueName: \"kubernetes.io/projected/eb60c02f-e91d-4b98-a61c-2bfe455bbe31-kube-api-access-nbpl5\") pod \"maas-api-e2e-derive-f22ea10f-5b5f999f7f-jq52n\" (UID: \"eb60c02f-e91d-4b98-a61c-2bfe455bbe31\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f22ea10f-5b5f999f7f-jq52n"
Jul 26 16:19:35.273052 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:35.272956    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb60c02f-e91d-4b98-a61c-2bfe455bbe31-openshift-service-ca\") pod \"maas-api-e2e-derive-f22ea10f-5b5f999f7f-jq52n\" (UID: \"eb60c02f-e91d-4b98-a61c-2bfe455bbe31\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f22ea10f-5b5f999f7f-jq52n"
Jul 26 16:19:35.315082 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:35.315050    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:19:35.374041 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:35.374002    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb60c02f-e91d-4b98-a61c-2bfe455bbe31-maas-api-tls\") pod \"maas-api-e2e-derive-f22ea10f-5b5f999f7f-jq52n\" (UID: \"eb60c02f-e91d-4b98-a61c-2bfe455bbe31\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f22ea10f-5b5f999f7f-jq52n"
Jul 26 16:19:35.374041 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:35.374046    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbpl5\" (UniqueName: \"kubernetes.io/projected/eb60c02f-e91d-4b98-a61c-2bfe455bbe31-kube-api-access-nbpl5\") pod \"maas-api-e2e-derive-f22ea10f-5b5f999f7f-jq52n\" (UID: \"eb60c02f-e91d-4b98-a61c-2bfe455bbe31\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f22ea10f-5b5f999f7f-jq52n"
Jul 26 16:19:35.374294 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:35.374091    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb60c02f-e91d-4b98-a61c-2bfe455bbe31-openshift-service-ca\") pod \"maas-api-e2e-derive-f22ea10f-5b5f999f7f-jq52n\" (UID: \"eb60c02f-e91d-4b98-a61c-2bfe455bbe31\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f22ea10f-5b5f999f7f-jq52n"
Jul 26 16:19:35.374294 ip-10-0-133-204 kubenswrapper[2563]: E0726 16:19:35.374146    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-f22ea10f-serving-cert: secret "maas-api-e2e-derive-f22ea10f-serving-cert" not found
Jul 26 16:19:35.374294 ip-10-0-133-204 kubenswrapper[2563]: E0726 16:19:35.374203    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb60c02f-e91d-4b98-a61c-2bfe455bbe31-maas-api-tls podName:eb60c02f-e91d-4b98-a61c-2bfe455bbe31 nodeName:}" failed. No retries permitted until 2026-07-26 16:19:35.874187501 +0000 UTC m=+1767.978413288 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/eb60c02f-e91d-4b98-a61c-2bfe455bbe31-maas-api-tls") pod "maas-api-e2e-derive-f22ea10f-5b5f999f7f-jq52n" (UID: "eb60c02f-e91d-4b98-a61c-2bfe455bbe31") : secret "maas-api-e2e-derive-f22ea10f-serving-cert" not found
Jul 26 16:19:35.374799 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:35.374752    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb60c02f-e91d-4b98-a61c-2bfe455bbe31-openshift-service-ca\") pod \"maas-api-e2e-derive-f22ea10f-5b5f999f7f-jq52n\" (UID: \"eb60c02f-e91d-4b98-a61c-2bfe455bbe31\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f22ea10f-5b5f999f7f-jq52n"
Jul 26 16:19:35.384132 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:35.384102    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbpl5\" (UniqueName: \"kubernetes.io/projected/eb60c02f-e91d-4b98-a61c-2bfe455bbe31-kube-api-access-nbpl5\") pod \"maas-api-e2e-derive-f22ea10f-5b5f999f7f-jq52n\" (UID: \"eb60c02f-e91d-4b98-a61c-2bfe455bbe31\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f22ea10f-5b5f999f7f-jq52n"
Jul 26 16:19:35.478942 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:35.478906    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:19:35.879070 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:35.878970    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb60c02f-e91d-4b98-a61c-2bfe455bbe31-maas-api-tls\") pod \"maas-api-e2e-derive-f22ea10f-5b5f999f7f-jq52n\" (UID: \"eb60c02f-e91d-4b98-a61c-2bfe455bbe31\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f22ea10f-5b5f999f7f-jq52n"
Jul 26 16:19:35.881729 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:35.881688    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb60c02f-e91d-4b98-a61c-2bfe455bbe31-maas-api-tls\") pod \"maas-api-e2e-derive-f22ea10f-5b5f999f7f-jq52n\" (UID: \"eb60c02f-e91d-4b98-a61c-2bfe455bbe31\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f22ea10f-5b5f999f7f-jq52n"
Jul 26 16:19:35.910748 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:35.910711    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:19:36.022420 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:36.022392    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:19:36.069363 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:36.069329    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f22ea10f-7d88b75dd6-p74pm"]
Jul 26 16:19:36.073000 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:36.072981    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-f22ea10f-7d88b75dd6-p74pm"
Jul 26 16:19:36.076444 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:36.076426    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-f22ea10f\""
Jul 26 16:19:36.077121 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:36.077106    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-f22ea10f-dockercfg-8nql4\""
Jul 26 16:19:36.080805 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:36.080784    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/602e70a1-49c2-49b6-a876-3817dc30e96d-plugins-config-volume\") pod \"payload-processing-e2e-derive-f22ea10f-7d88b75dd6-p74pm\" (UID: \"602e70a1-49c2-49b6-a876-3817dc30e96d\") " pod="openshift-ingress/payload-processing-e2e-derive-f22ea10f-7d88b75dd6-p74pm"
Jul 26 16:19:36.080886 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:36.080824    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rcfr\" (UniqueName: \"kubernetes.io/projected/602e70a1-49c2-49b6-a876-3817dc30e96d-kube-api-access-6rcfr\") pod \"payload-processing-e2e-derive-f22ea10f-7d88b75dd6-p74pm\" (UID: \"602e70a1-49c2-49b6-a876-3817dc30e96d\") " pod="openshift-ingress/payload-processing-e2e-derive-f22ea10f-7d88b75dd6-p74pm"
Jul 26 16:19:36.096901 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:36.096874    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f22ea10f-7d88b75dd6-p74pm"]
Jul 26 16:19:36.145057 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:36.144971    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-f22ea10f-6675769cfb-l5xs8"]
Jul 26 16:19:36.148511 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:36.148493    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-f22ea10f-6675769cfb-l5xs8"
Jul 26 16:19:36.166924 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:36.166888    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f22ea10f-5b5f999f7f-jq52n"
Jul 26 16:19:36.182267 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:36.181644    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/602e70a1-49c2-49b6-a876-3817dc30e96d-plugins-config-volume\") pod \"payload-processing-e2e-derive-f22ea10f-7d88b75dd6-p74pm\" (UID: \"602e70a1-49c2-49b6-a876-3817dc30e96d\") " pod="openshift-ingress/payload-processing-e2e-derive-f22ea10f-7d88b75dd6-p74pm"
Jul 26 16:19:36.182267 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:36.181712    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rcfr\" (UniqueName: \"kubernetes.io/projected/602e70a1-49c2-49b6-a876-3817dc30e96d-kube-api-access-6rcfr\") pod \"payload-processing-e2e-derive-f22ea10f-7d88b75dd6-p74pm\" (UID: \"602e70a1-49c2-49b6-a876-3817dc30e96d\") " pod="openshift-ingress/payload-processing-e2e-derive-f22ea10f-7d88b75dd6-p74pm"
Jul 26 16:19:36.182267 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:36.181776    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvssc\" (UniqueName: \"kubernetes.io/projected/7d99a295-25e0-4857-b597-0ee3ef5224fa-kube-api-access-rvssc\") pod \"payload-pre-processing-e2e-derive-f22ea10f-6675769cfb-l5xs8\" (UID: \"7d99a295-25e0-4857-b597-0ee3ef5224fa\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-f22ea10f-6675769cfb-l5xs8"
Jul 26 16:19:36.182267 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:36.181856    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d99a295-25e0-4857-b597-0ee3ef5224fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-f22ea10f-6675769cfb-l5xs8\" (UID: \"7d99a295-25e0-4857-b597-0ee3ef5224fa\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-f22ea10f-6675769cfb-l5xs8"
Jul 26 16:19:36.182591 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:36.182538    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-f22ea10f-6675769cfb-l5xs8"]
Jul 26 16:19:36.182591 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:36.182559    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/602e70a1-49c2-49b6-a876-3817dc30e96d-plugins-config-volume\") pod \"payload-processing-e2e-derive-f22ea10f-7d88b75dd6-p74pm\" (UID: \"602e70a1-49c2-49b6-a876-3817dc30e96d\") " pod="openshift-ingress/payload-processing-e2e-derive-f22ea10f-7d88b75dd6-p74pm"
Jul 26 16:19:36.193857 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:36.193743    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rcfr\" (UniqueName: \"kubernetes.io/projected/602e70a1-49c2-49b6-a876-3817dc30e96d-kube-api-access-6rcfr\") pod \"payload-processing-e2e-derive-f22ea10f-7d88b75dd6-p74pm\" (UID: \"602e70a1-49c2-49b6-a876-3817dc30e96d\") " pod="openshift-ingress/payload-processing-e2e-derive-f22ea10f-7d88b75dd6-p74pm"
Jul 26 16:19:36.282642 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:36.282606    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvssc\" (UniqueName: \"kubernetes.io/projected/7d99a295-25e0-4857-b597-0ee3ef5224fa-kube-api-access-rvssc\") pod \"payload-pre-processing-e2e-derive-f22ea10f-6675769cfb-l5xs8\" (UID: \"7d99a295-25e0-4857-b597-0ee3ef5224fa\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-f22ea10f-6675769cfb-l5xs8"
Jul 26 16:19:36.282814 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:36.282669    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d99a295-25e0-4857-b597-0ee3ef5224fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-f22ea10f-6675769cfb-l5xs8\" (UID: \"7d99a295-25e0-4857-b597-0ee3ef5224fa\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-f22ea10f-6675769cfb-l5xs8"
Jul 26 16:19:36.283942 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:36.283898    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d99a295-25e0-4857-b597-0ee3ef5224fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-f22ea10f-6675769cfb-l5xs8\" (UID: \"7d99a295-25e0-4857-b597-0ee3ef5224fa\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-f22ea10f-6675769cfb-l5xs8"
Jul 26 16:19:36.292645 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:36.292610    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvssc\" (UniqueName: \"kubernetes.io/projected/7d99a295-25e0-4857-b597-0ee3ef5224fa-kube-api-access-rvssc\") pod \"payload-pre-processing-e2e-derive-f22ea10f-6675769cfb-l5xs8\" (UID: \"7d99a295-25e0-4857-b597-0ee3ef5224fa\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-f22ea10f-6675769cfb-l5xs8"
Jul 26 16:19:36.383834 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:36.383799    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-f22ea10f-7d88b75dd6-p74pm"
Jul 26 16:19:36.459154 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:36.459126    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-f22ea10f-6675769cfb-l5xs8"
Jul 26 16:19:36.528663 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:36.528601    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f22ea10f-5b5f999f7f-jq52n"]
Jul 26 16:19:36.533576 ip-10-0-133-204 kubenswrapper[2563]: W0726 16:19:36.533515    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb60c02f_e91d_4b98_a61c_2bfe455bbe31.slice/crio-f6dc7655c95379041e11edf3dc8e0991259998ad124f1fdff6a24ec648f29815 WatchSource:0}: Error finding container f6dc7655c95379041e11edf3dc8e0991259998ad124f1fdff6a24ec648f29815: Status 404 returned error can't find the container with id f6dc7655c95379041e11edf3dc8e0991259998ad124f1fdff6a24ec648f29815
Jul 26 16:19:36.541808 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:36.541748    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f22ea10f-7d88b75dd6-p74pm"]
Jul 26 16:19:36.543989 ip-10-0-133-204 kubenswrapper[2563]: W0726 16:19:36.543960    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod602e70a1_49c2_49b6_a876_3817dc30e96d.slice/crio-a634f990490672056b8ed82f2331b92a7bc632d3a42d57e68be019b28767753d WatchSource:0}: Error finding container a634f990490672056b8ed82f2331b92a7bc632d3a42d57e68be019b28767753d: Status 404 returned error can't find the container with id a634f990490672056b8ed82f2331b92a7bc632d3a42d57e68be019b28767753d
Jul 26 16:19:36.625033 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:36.624998    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-f22ea10f-6675769cfb-l5xs8"]
Jul 26 16:19:36.628632 ip-10-0-133-204 kubenswrapper[2563]: W0726 16:19:36.628597    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d99a295_25e0_4857_b597_0ee3ef5224fa.slice/crio-fb97c550e52951c31cc818ba26726f2d0f11964e3570071f10ae843fd8dcde67 WatchSource:0}: Error finding container fb97c550e52951c31cc818ba26726f2d0f11964e3570071f10ae843fd8dcde67: Status 404 returned error can't find the container with id fb97c550e52951c31cc818ba26726f2d0f11964e3570071f10ae843fd8dcde67
Jul 26 16:19:36.930129 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:36.930045    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-f22ea10f-7d88b75dd6-p74pm" event={"ID":"602e70a1-49c2-49b6-a876-3817dc30e96d","Type":"ContainerStarted","Data":"8328d6bb3e143b4f7a78b824fda6fae8336c072bf58a21e166653811ecaac2a9"}
Jul 26 16:19:36.930129 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:36.930128    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-f22ea10f-7d88b75dd6-p74pm" event={"ID":"602e70a1-49c2-49b6-a876-3817dc30e96d","Type":"ContainerStarted","Data":"a634f990490672056b8ed82f2331b92a7bc632d3a42d57e68be019b28767753d"}
Jul 26 16:19:36.930363 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:36.930165    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-f22ea10f-7d88b75dd6-p74pm"
Jul 26 16:19:36.931372 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:36.931340    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f22ea10f-5b5f999f7f-jq52n" event={"ID":"eb60c02f-e91d-4b98-a61c-2bfe455bbe31","Type":"ContainerStarted","Data":"f6dc7655c95379041e11edf3dc8e0991259998ad124f1fdff6a24ec648f29815"}
Jul 26 16:19:36.933136 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:36.933114    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-f22ea10f-6675769cfb-l5xs8" event={"ID":"7d99a295-25e0-4857-b597-0ee3ef5224fa","Type":"ContainerStarted","Data":"6b72461989ff41e88d9e5e3ed58f5c1a01b93bbbc7811db7a4daba34c6048820"}
Jul 26 16:19:36.933136 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:36.933157    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-f22ea10f-6675769cfb-l5xs8" event={"ID":"7d99a295-25e0-4857-b597-0ee3ef5224fa","Type":"ContainerStarted","Data":"fb97c550e52951c31cc818ba26726f2d0f11964e3570071f10ae843fd8dcde67"}
Jul 26 16:19:36.933330 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:36.933198    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-f22ea10f-6675769cfb-l5xs8"
Jul 26 16:19:36.945844 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:36.945659    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-f22ea10f-7d88b75dd6-p74pm" podStartSLOduration=0.945640971 podStartE2EDuration="945.640971ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:19:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:19:36.945065592 +0000 UTC m=+1769.049291401" watchObservedRunningTime="2026-07-26 16:19:36.945640971 +0000 UTC m=+1769.049866781"
Jul 26 16:19:36.961308 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:36.961250    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-f22ea10f-6675769cfb-l5xs8" podStartSLOduration=0.961228313 podStartE2EDuration="961.228313ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:19:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:19:36.959670746 +0000 UTC m=+1769.063896556" watchObservedRunningTime="2026-07-26 16:19:36.961228313 +0000 UTC m=+1769.065454122"
Jul 26 16:19:37.939652 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:37.939616    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f22ea10f-5b5f999f7f-jq52n" event={"ID":"eb60c02f-e91d-4b98-a61c-2bfe455bbe31","Type":"ContainerStarted","Data":"3dd839645c5f1c3f6b48de8c5e10402cf6b1351a040027a9c77020bbc7f40da0"}
Jul 26 16:19:37.940139 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:37.939863    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f22ea10f-5b5f999f7f-jq52n"
Jul 26 16:19:37.957352 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:37.957303    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f22ea10f-5b5f999f7f-jq52n" podStartSLOduration=2.560637122 podStartE2EDuration="2.957286635s" totalImagesPullingTime="396.649513ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:19:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:19:37.955450788 +0000 UTC m=+1770.059676612" watchObservedRunningTime="2026-07-26 16:19:37.957286635 +0000 UTC m=+1770.061512444"
Jul 26 16:19:43.950364 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:43.950330    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f22ea10f-5b5f999f7f-jq52n"
Jul 26 16:19:47.942174 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:47.942141    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-f22ea10f-7d88b75dd6-p74pm"
Jul 26 16:19:47.942623 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:19:47.942210    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-f22ea10f-6675769cfb-l5xs8"
Jul 26 16:20:10.509571 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:10.509535    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f22ea10f-5b5f999f7f-jq52n"]
Jul 26 16:20:10.510102 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:10.509751    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f22ea10f-5b5f999f7f-jq52n" podUID="eb60c02f-e91d-4b98-a61c-2bfe455bbe31" containerName="maas-api" containerID="cri-o://3dd839645c5f1c3f6b48de8c5e10402cf6b1351a040027a9c77020bbc7f40da0" gracePeriod=30
Jul 26 16:20:10.526645 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:10.526619    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:20:10.620414 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:10.620379    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f22ea10f-7d88b75dd6-p74pm"]
Jul 26 16:20:10.620675 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:10.620645    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-f22ea10f-7d88b75dd6-p74pm" podUID="602e70a1-49c2-49b6-a876-3817dc30e96d" containerName="payload-processing" containerID="cri-o://8328d6bb3e143b4f7a78b824fda6fae8336c072bf58a21e166653811ecaac2a9" gracePeriod=30
Jul 26 16:20:10.631820 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:10.631788    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-f22ea10f-6675769cfb-l5xs8"]
Jul 26 16:20:10.632078 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:10.632051    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-f22ea10f-6675769cfb-l5xs8" podUID="7d99a295-25e0-4857-b597-0ee3ef5224fa" containerName="payload-pre-processing" containerID="cri-o://6b72461989ff41e88d9e5e3ed58f5c1a01b93bbbc7811db7a4daba34c6048820" gracePeriod=30
Jul 26 16:20:10.671457 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:10.671427    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:20:10.772143 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:10.772030    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:20:10.788500 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:10.788199    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f22ea10f-5b5f999f7f-jq52n"
Jul 26 16:20:10.816235 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:10.816192    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb60c02f-e91d-4b98-a61c-2bfe455bbe31-maas-api-tls\") pod \"eb60c02f-e91d-4b98-a61c-2bfe455bbe31\" (UID: \"eb60c02f-e91d-4b98-a61c-2bfe455bbe31\") "
Jul 26 16:20:10.816235 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:10.816236    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nbpl5\" (UniqueName: \"kubernetes.io/projected/eb60c02f-e91d-4b98-a61c-2bfe455bbe31-kube-api-access-nbpl5\") pod \"eb60c02f-e91d-4b98-a61c-2bfe455bbe31\" (UID: \"eb60c02f-e91d-4b98-a61c-2bfe455bbe31\") "
Jul 26 16:20:10.816532 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:10.816355    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb60c02f-e91d-4b98-a61c-2bfe455bbe31-openshift-service-ca\") pod \"eb60c02f-e91d-4b98-a61c-2bfe455bbe31\" (UID: \"eb60c02f-e91d-4b98-a61c-2bfe455bbe31\") "
Jul 26 16:20:10.817097 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:10.817066    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb60c02f-e91d-4b98-a61c-2bfe455bbe31-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "eb60c02f-e91d-4b98-a61c-2bfe455bbe31" (UID: "eb60c02f-e91d-4b98-a61c-2bfe455bbe31"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 26 16:20:10.819146 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:10.819119    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb60c02f-e91d-4b98-a61c-2bfe455bbe31-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "eb60c02f-e91d-4b98-a61c-2bfe455bbe31" (UID: "eb60c02f-e91d-4b98-a61c-2bfe455bbe31"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 26 16:20:10.819242 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:10.819141    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb60c02f-e91d-4b98-a61c-2bfe455bbe31-kube-api-access-nbpl5" (OuterVolumeSpecName: "kube-api-access-nbpl5") pod "eb60c02f-e91d-4b98-a61c-2bfe455bbe31" (UID: "eb60c02f-e91d-4b98-a61c-2bfe455bbe31"). InnerVolumeSpecName "kube-api-access-nbpl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 26 16:20:10.880425 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:10.880400    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-f22ea10f-7d88b75dd6-p74pm"
Jul 26 16:20:10.917009 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:10.916966    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6rcfr\" (UniqueName: \"kubernetes.io/projected/602e70a1-49c2-49b6-a876-3817dc30e96d-kube-api-access-6rcfr\") pod \"602e70a1-49c2-49b6-a876-3817dc30e96d\" (UID: \"602e70a1-49c2-49b6-a876-3817dc30e96d\") "
Jul 26 16:20:10.917184 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:10.917117    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/602e70a1-49c2-49b6-a876-3817dc30e96d-plugins-config-volume\") pod \"602e70a1-49c2-49b6-a876-3817dc30e96d\" (UID: \"602e70a1-49c2-49b6-a876-3817dc30e96d\") "
Jul 26 16:20:10.917402 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:10.917385    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb60c02f-e91d-4b98-a61c-2bfe455bbe31-maas-api-tls\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:20:10.917479 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:10.917403    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbpl5\" (UniqueName: \"kubernetes.io/projected/eb60c02f-e91d-4b98-a61c-2bfe455bbe31-kube-api-access-nbpl5\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:20:10.917479 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:10.917412    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb60c02f-e91d-4b98-a61c-2bfe455bbe31-openshift-service-ca\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:20:10.917479 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:10.917453    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/602e70a1-49c2-49b6-a876-3817dc30e96d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "602e70a1-49c2-49b6-a876-3817dc30e96d" (UID: "602e70a1-49c2-49b6-a876-3817dc30e96d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 26 16:20:10.919228 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:10.919196    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/602e70a1-49c2-49b6-a876-3817dc30e96d-kube-api-access-6rcfr" (OuterVolumeSpecName: "kube-api-access-6rcfr") pod "602e70a1-49c2-49b6-a876-3817dc30e96d" (UID: "602e70a1-49c2-49b6-a876-3817dc30e96d"). InnerVolumeSpecName "kube-api-access-6rcfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 26 16:20:10.919373 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:10.919357    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-f22ea10f-6675769cfb-l5xs8"
Jul 26 16:20:11.018512 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:11.018433    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rvssc\" (UniqueName: \"kubernetes.io/projected/7d99a295-25e0-4857-b597-0ee3ef5224fa-kube-api-access-rvssc\") pod \"7d99a295-25e0-4857-b597-0ee3ef5224fa\" (UID: \"7d99a295-25e0-4857-b597-0ee3ef5224fa\") "
Jul 26 16:20:11.018512 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:11.018510    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d99a295-25e0-4857-b597-0ee3ef5224fa-plugins-config-volume\") pod \"7d99a295-25e0-4857-b597-0ee3ef5224fa\" (UID: \"7d99a295-25e0-4857-b597-0ee3ef5224fa\") "
Jul 26 16:20:11.018708 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:11.018666    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/602e70a1-49c2-49b6-a876-3817dc30e96d-plugins-config-volume\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:20:11.018708 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:11.018676    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rcfr\" (UniqueName: \"kubernetes.io/projected/602e70a1-49c2-49b6-a876-3817dc30e96d-kube-api-access-6rcfr\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:20:11.018913 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:11.018884    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d99a295-25e0-4857-b597-0ee3ef5224fa-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7d99a295-25e0-4857-b597-0ee3ef5224fa" (UID: "7d99a295-25e0-4857-b597-0ee3ef5224fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 26 16:20:11.020502 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:11.020473    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d99a295-25e0-4857-b597-0ee3ef5224fa-kube-api-access-rvssc" (OuterVolumeSpecName: "kube-api-access-rvssc") pod "7d99a295-25e0-4857-b597-0ee3ef5224fa" (UID: "7d99a295-25e0-4857-b597-0ee3ef5224fa"). InnerVolumeSpecName "kube-api-access-rvssc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 26 16:20:11.107689 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:11.107652    2563 generic.go:356] "Generic (PLEG): container finished" podID="602e70a1-49c2-49b6-a876-3817dc30e96d" containerID="8328d6bb3e143b4f7a78b824fda6fae8336c072bf58a21e166653811ecaac2a9" exitCode=0
Jul 26 16:20:11.107885 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:11.107697    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-f22ea10f-7d88b75dd6-p74pm"
Jul 26 16:20:11.107885 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:11.107744    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-f22ea10f-7d88b75dd6-p74pm" event={"ID":"602e70a1-49c2-49b6-a876-3817dc30e96d","Type":"ContainerDied","Data":"8328d6bb3e143b4f7a78b824fda6fae8336c072bf58a21e166653811ecaac2a9"}
Jul 26 16:20:11.107885 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:11.107836    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-f22ea10f-7d88b75dd6-p74pm" event={"ID":"602e70a1-49c2-49b6-a876-3817dc30e96d","Type":"ContainerDied","Data":"a634f990490672056b8ed82f2331b92a7bc632d3a42d57e68be019b28767753d"}
Jul 26 16:20:11.107885 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:11.107871    2563 scope.go:117] "RemoveContainer" containerID="8328d6bb3e143b4f7a78b824fda6fae8336c072bf58a21e166653811ecaac2a9"
Jul 26 16:20:11.109099 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:11.109079    2563 generic.go:356] "Generic (PLEG): container finished" podID="eb60c02f-e91d-4b98-a61c-2bfe455bbe31" containerID="3dd839645c5f1c3f6b48de8c5e10402cf6b1351a040027a9c77020bbc7f40da0" exitCode=0
Jul 26 16:20:11.109188 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:11.109122    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f22ea10f-5b5f999f7f-jq52n"
Jul 26 16:20:11.109188 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:11.109169    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f22ea10f-5b5f999f7f-jq52n" event={"ID":"eb60c02f-e91d-4b98-a61c-2bfe455bbe31","Type":"ContainerDied","Data":"3dd839645c5f1c3f6b48de8c5e10402cf6b1351a040027a9c77020bbc7f40da0"}
Jul 26 16:20:11.109292 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:11.109219    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f22ea10f-5b5f999f7f-jq52n" event={"ID":"eb60c02f-e91d-4b98-a61c-2bfe455bbe31","Type":"ContainerDied","Data":"f6dc7655c95379041e11edf3dc8e0991259998ad124f1fdff6a24ec648f29815"}
Jul 26 16:20:11.110648 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:11.110622    2563 generic.go:356] "Generic (PLEG): container finished" podID="7d99a295-25e0-4857-b597-0ee3ef5224fa" containerID="6b72461989ff41e88d9e5e3ed58f5c1a01b93bbbc7811db7a4daba34c6048820" exitCode=0
Jul 26 16:20:11.110756 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:11.110652    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-f22ea10f-6675769cfb-l5xs8"
Jul 26 16:20:11.110756 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:11.110700    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-f22ea10f-6675769cfb-l5xs8" event={"ID":"7d99a295-25e0-4857-b597-0ee3ef5224fa","Type":"ContainerDied","Data":"6b72461989ff41e88d9e5e3ed58f5c1a01b93bbbc7811db7a4daba34c6048820"}
Jul 26 16:20:11.110756 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:11.110735    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-f22ea10f-6675769cfb-l5xs8" event={"ID":"7d99a295-25e0-4857-b597-0ee3ef5224fa","Type":"ContainerDied","Data":"fb97c550e52951c31cc818ba26726f2d0f11964e3570071f10ae843fd8dcde67"}
Jul 26 16:20:11.118587 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:11.118563    2563 scope.go:117] "RemoveContainer" containerID="8328d6bb3e143b4f7a78b824fda6fae8336c072bf58a21e166653811ecaac2a9"
Jul 26 16:20:11.118935 ip-10-0-133-204 kubenswrapper[2563]: E0726 16:20:11.118910    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8328d6bb3e143b4f7a78b824fda6fae8336c072bf58a21e166653811ecaac2a9\": container with ID starting with 8328d6bb3e143b4f7a78b824fda6fae8336c072bf58a21e166653811ecaac2a9 not found: ID does not exist" containerID="8328d6bb3e143b4f7a78b824fda6fae8336c072bf58a21e166653811ecaac2a9"
Jul 26 16:20:11.119097 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:11.118943    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8328d6bb3e143b4f7a78b824fda6fae8336c072bf58a21e166653811ecaac2a9"} err="failed to get container status \"8328d6bb3e143b4f7a78b824fda6fae8336c072bf58a21e166653811ecaac2a9\": rpc error: code = NotFound desc = could not find container \"8328d6bb3e143b4f7a78b824fda6fae8336c072bf58a21e166653811ecaac2a9\": container with ID starting with 8328d6bb3e143b4f7a78b824fda6fae8336c072bf58a21e166653811ecaac2a9 not found: ID does not exist"
Jul 26 16:20:11.119097 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:11.118965    2563 scope.go:117] "RemoveContainer" containerID="3dd839645c5f1c3f6b48de8c5e10402cf6b1351a040027a9c77020bbc7f40da0"
Jul 26 16:20:11.119276 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:11.119249    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d99a295-25e0-4857-b597-0ee3ef5224fa-plugins-config-volume\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:20:11.119389 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:11.119279    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvssc\" (UniqueName: \"kubernetes.io/projected/7d99a295-25e0-4857-b597-0ee3ef5224fa-kube-api-access-rvssc\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:20:11.128111 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:11.128090    2563 scope.go:117] "RemoveContainer" containerID="3dd839645c5f1c3f6b48de8c5e10402cf6b1351a040027a9c77020bbc7f40da0"
Jul 26 16:20:11.128431 ip-10-0-133-204 kubenswrapper[2563]: E0726 16:20:11.128410    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dd839645c5f1c3f6b48de8c5e10402cf6b1351a040027a9c77020bbc7f40da0\": container with ID starting with 3dd839645c5f1c3f6b48de8c5e10402cf6b1351a040027a9c77020bbc7f40da0 not found: ID does not exist" containerID="3dd839645c5f1c3f6b48de8c5e10402cf6b1351a040027a9c77020bbc7f40da0"
Jul 26 16:20:11.128522 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:11.128437    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dd839645c5f1c3f6b48de8c5e10402cf6b1351a040027a9c77020bbc7f40da0"} err="failed to get container status \"3dd839645c5f1c3f6b48de8c5e10402cf6b1351a040027a9c77020bbc7f40da0\": rpc error: code = NotFound desc = could not find container \"3dd839645c5f1c3f6b48de8c5e10402cf6b1351a040027a9c77020bbc7f40da0\": container with ID starting with 3dd839645c5f1c3f6b48de8c5e10402cf6b1351a040027a9c77020bbc7f40da0 not found: ID does not exist"
Jul 26 16:20:11.128522 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:11.128467    2563 scope.go:117] "RemoveContainer" containerID="6b72461989ff41e88d9e5e3ed58f5c1a01b93bbbc7811db7a4daba34c6048820"
Jul 26 16:20:11.131498 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:11.131478    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f22ea10f-7d88b75dd6-p74pm"]
Jul 26 16:20:11.137656 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:11.137637    2563 scope.go:117] "RemoveContainer" containerID="6b72461989ff41e88d9e5e3ed58f5c1a01b93bbbc7811db7a4daba34c6048820"
Jul 26 16:20:11.138090 ip-10-0-133-204 kubenswrapper[2563]: E0726 16:20:11.138067    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b72461989ff41e88d9e5e3ed58f5c1a01b93bbbc7811db7a4daba34c6048820\": container with ID starting with 6b72461989ff41e88d9e5e3ed58f5c1a01b93bbbc7811db7a4daba34c6048820 not found: ID does not exist" containerID="6b72461989ff41e88d9e5e3ed58f5c1a01b93bbbc7811db7a4daba34c6048820"
Jul 26 16:20:11.138184 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:11.138098    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b72461989ff41e88d9e5e3ed58f5c1a01b93bbbc7811db7a4daba34c6048820"} err="failed to get container status \"6b72461989ff41e88d9e5e3ed58f5c1a01b93bbbc7811db7a4daba34c6048820\": rpc error: code = NotFound desc = could not find container \"6b72461989ff41e88d9e5e3ed58f5c1a01b93bbbc7811db7a4daba34c6048820\": container with ID starting with 6b72461989ff41e88d9e5e3ed58f5c1a01b93bbbc7811db7a4daba34c6048820 not found: ID does not exist"
Jul 26 16:20:11.139221 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:11.139200    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f22ea10f-7d88b75dd6-p74pm"]
Jul 26 16:20:11.150217 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:11.150186    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f22ea10f-5b5f999f7f-jq52n"]
Jul 26 16:20:11.153519 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:11.153496    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f22ea10f-5b5f999f7f-jq52n"]
Jul 26 16:20:11.164426 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:11.164401    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-f22ea10f-6675769cfb-l5xs8"]
Jul 26 16:20:11.167960 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:11.167938    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-f22ea10f-6675769cfb-l5xs8"]
Jul 26 16:20:12.372727 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:12.372685    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="602e70a1-49c2-49b6-a876-3817dc30e96d" path="/var/lib/kubelet/pods/602e70a1-49c2-49b6-a876-3817dc30e96d/volumes"
Jul 26 16:20:12.373205 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:12.373068    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d99a295-25e0-4857-b597-0ee3ef5224fa" path="/var/lib/kubelet/pods/7d99a295-25e0-4857-b597-0ee3ef5224fa/volumes"
Jul 26 16:20:12.373405 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:12.373390    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb60c02f-e91d-4b98-a61c-2bfe455bbe31" path="/var/lib/kubelet/pods/eb60c02f-e91d-4b98-a61c-2bfe455bbe31/volumes"
Jul 26 16:20:15.026438 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:15.026406    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:20:15.183639 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:15.183605    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:20:15.419461 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:15.419360    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:20:15.916888 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:15.916839    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:20:16.020733 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:16.020689    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:20:16.118391 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:16.118358    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:20:17.589917 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:17.589874    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:20:20.415956 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:20.415925    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:20:23.276261 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:23.276222    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:20:23.382014 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:23.381980    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:20:23.476354 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:23.476313    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:20:23.576161 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:23.576064    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:20:23.985161 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:23.985132    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:20:49.978009 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:49.977979    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:20:56.248701 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:56.248664    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:20:56.355603 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:56.355571    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:20:56.451688 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:20:56.451658    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:21:12.389368 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:12.389335    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:21:18.212436 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:18.212404    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:21:18.278181 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:18.278150    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:21:18.366327 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:18.366290    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:21:18.469094 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:18.469013    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:21:28.952011 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:28.951972    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:21:29.799076 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:29.799037    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:21:36.243491 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:36.243455    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:21:36.349128 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:36.349096    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:21:36.444666 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:36.444631    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:21:36.544390 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:36.544289    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:21:37.376634 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:37.376598    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:21:37.465584 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:37.465554    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:21:37.566299 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:37.566272    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:21:39.306819 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:39.306778    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:21:45.771641 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:45.771606    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:21:45.871741 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:45.871711    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:21:46.487665 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:46.487629    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:21:52.350960 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:52.350926    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:21:52.414547 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:52.414513    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:21:52.599521 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:52.599489    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:21:52.953237 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:52.953195    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:21:53.207723 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:53.207631    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:21:55.775593 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:55.775555    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:21:55.878735 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:55.878696    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:21:55.970729 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:55.970699    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6d044d53-786985db86-78b87"]
Jul 26 16:21:55.971125 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:55.971110    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb60c02f-e91d-4b98-a61c-2bfe455bbe31" containerName="maas-api"
Jul 26 16:21:55.971170 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:55.971126    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb60c02f-e91d-4b98-a61c-2bfe455bbe31" containerName="maas-api"
Jul 26 16:21:55.971170 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:55.971137    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d99a295-25e0-4857-b597-0ee3ef5224fa" containerName="payload-pre-processing"
Jul 26 16:21:55.971170 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:55.971143    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d99a295-25e0-4857-b597-0ee3ef5224fa" containerName="payload-pre-processing"
Jul 26 16:21:55.971271 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:55.971173    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="602e70a1-49c2-49b6-a876-3817dc30e96d" containerName="payload-processing"
Jul 26 16:21:55.971271 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:55.971179    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="602e70a1-49c2-49b6-a876-3817dc30e96d" containerName="payload-processing"
Jul 26 16:21:55.971271 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:55.971240    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="602e70a1-49c2-49b6-a876-3817dc30e96d" containerName="payload-processing"
Jul 26 16:21:55.971271 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:55.971254    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d99a295-25e0-4857-b597-0ee3ef5224fa" containerName="payload-pre-processing"
Jul 26 16:21:55.971271 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:55.971262    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb60c02f-e91d-4b98-a61c-2bfe455bbe31" containerName="maas-api"
Jul 26 16:21:55.974546 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:55.974522    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d044d53-786985db86-78b87"
Jul 26 16:21:55.977140 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:55.977116    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-6d044d53-serving-cert\""
Jul 26 16:21:55.989930 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:55.989902    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6d044d53-786985db86-78b87"]
Jul 26 16:21:56.015171 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:56.015135    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:21:56.107872 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:56.107753    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcd9528e-ba1a-4260-bba6-589e5377cba4-openshift-service-ca\") pod \"maas-api-e2e-mt-6d044d53-786985db86-78b87\" (UID: \"dcd9528e-ba1a-4260-bba6-589e5377cba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d044d53-786985db86-78b87"
Jul 26 16:21:56.107872 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:56.107829    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45h7g\" (UniqueName: \"kubernetes.io/projected/dcd9528e-ba1a-4260-bba6-589e5377cba4-kube-api-access-45h7g\") pod \"maas-api-e2e-mt-6d044d53-786985db86-78b87\" (UID: \"dcd9528e-ba1a-4260-bba6-589e5377cba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d044d53-786985db86-78b87"
Jul 26 16:21:56.108089 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:56.107906    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dcd9528e-ba1a-4260-bba6-589e5377cba4-maas-api-tls\") pod \"maas-api-e2e-mt-6d044d53-786985db86-78b87\" (UID: \"dcd9528e-ba1a-4260-bba6-589e5377cba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d044d53-786985db86-78b87"
Jul 26 16:21:56.208643 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:56.208606    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dcd9528e-ba1a-4260-bba6-589e5377cba4-maas-api-tls\") pod \"maas-api-e2e-mt-6d044d53-786985db86-78b87\" (UID: \"dcd9528e-ba1a-4260-bba6-589e5377cba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d044d53-786985db86-78b87"
Jul 26 16:21:56.208848 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:56.208679    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcd9528e-ba1a-4260-bba6-589e5377cba4-openshift-service-ca\") pod \"maas-api-e2e-mt-6d044d53-786985db86-78b87\" (UID: \"dcd9528e-ba1a-4260-bba6-589e5377cba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d044d53-786985db86-78b87"
Jul 26 16:21:56.208848 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:56.208711    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45h7g\" (UniqueName: \"kubernetes.io/projected/dcd9528e-ba1a-4260-bba6-589e5377cba4-kube-api-access-45h7g\") pod \"maas-api-e2e-mt-6d044d53-786985db86-78b87\" (UID: \"dcd9528e-ba1a-4260-bba6-589e5377cba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d044d53-786985db86-78b87"
Jul 26 16:21:56.208848 ip-10-0-133-204 kubenswrapper[2563]: E0726 16:21:56.208777    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-6d044d53-serving-cert: secret "maas-api-e2e-mt-6d044d53-serving-cert" not found
Jul 26 16:21:56.208848 ip-10-0-133-204 kubenswrapper[2563]: E0726 16:21:56.208849    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcd9528e-ba1a-4260-bba6-589e5377cba4-maas-api-tls podName:dcd9528e-ba1a-4260-bba6-589e5377cba4 nodeName:}" failed. No retries permitted until 2026-07-26 16:21:56.708834686 +0000 UTC m=+1908.813060472 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dcd9528e-ba1a-4260-bba6-589e5377cba4-maas-api-tls") pod "maas-api-e2e-mt-6d044d53-786985db86-78b87" (UID: "dcd9528e-ba1a-4260-bba6-589e5377cba4") : secret "maas-api-e2e-mt-6d044d53-serving-cert" not found
Jul 26 16:21:56.209401 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:56.209380    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcd9528e-ba1a-4260-bba6-589e5377cba4-openshift-service-ca\") pod \"maas-api-e2e-mt-6d044d53-786985db86-78b87\" (UID: \"dcd9528e-ba1a-4260-bba6-589e5377cba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d044d53-786985db86-78b87"
Jul 26 16:21:56.220402 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:56.220370    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45h7g\" (UniqueName: \"kubernetes.io/projected/dcd9528e-ba1a-4260-bba6-589e5377cba4-kube-api-access-45h7g\") pod \"maas-api-e2e-mt-6d044d53-786985db86-78b87\" (UID: \"dcd9528e-ba1a-4260-bba6-589e5377cba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d044d53-786985db86-78b87"
Jul 26 16:21:56.400530 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:56.400436    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:21:56.540989 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:56.540958    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:21:56.646372 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:56.646340    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:21:56.647532 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:56.647510    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6d044d53-644c44b476-hgv7f"]
Jul 26 16:21:56.651481 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:56.651429    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6d044d53-644c44b476-hgv7f"
Jul 26 16:21:56.654075 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:56.654045    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-6d044d53\""
Jul 26 16:21:56.654188 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:56.654118    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-6d044d53-dockercfg-7fplj\""
Jul 26 16:21:56.663695 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:56.663670    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6d044d53-644c44b476-hgv7f"]
Jul 26 16:21:56.666364 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:56.666335    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6d044d53-5c7d778998-sqrmn"]
Jul 26 16:21:56.670614 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:56.670595    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6d044d53-5c7d778998-sqrmn"
Jul 26 16:21:56.682297 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:56.682273    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6d044d53-5c7d778998-sqrmn"]
Jul 26 16:21:56.712290 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:56.712260    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbrjp\" (UniqueName: \"kubernetes.io/projected/95afe73b-8654-4e36-aa88-b28b9ca338f2-kube-api-access-tbrjp\") pod \"payload-pre-processing-e2e-mt-6d044d53-644c44b476-hgv7f\" (UID: \"95afe73b-8654-4e36-aa88-b28b9ca338f2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6d044d53-644c44b476-hgv7f"
Jul 26 16:21:56.712452 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:56.712321    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95afe73b-8654-4e36-aa88-b28b9ca338f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6d044d53-644c44b476-hgv7f\" (UID: \"95afe73b-8654-4e36-aa88-b28b9ca338f2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6d044d53-644c44b476-hgv7f"
Jul 26 16:21:56.712452 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:56.712370    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dcd9528e-ba1a-4260-bba6-589e5377cba4-maas-api-tls\") pod \"maas-api-e2e-mt-6d044d53-786985db86-78b87\" (UID: \"dcd9528e-ba1a-4260-bba6-589e5377cba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d044d53-786985db86-78b87"
Jul 26 16:21:56.714695 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:56.714666    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dcd9528e-ba1a-4260-bba6-589e5377cba4-maas-api-tls\") pod \"maas-api-e2e-mt-6d044d53-786985db86-78b87\" (UID: \"dcd9528e-ba1a-4260-bba6-589e5377cba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d044d53-786985db86-78b87"
Jul 26 16:21:56.814008 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:56.813961    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcb8v\" (UniqueName: \"kubernetes.io/projected/3c4995b5-ff59-4d08-bff2-8f49d69619a9-kube-api-access-pcb8v\") pod \"payload-processing-e2e-mt-6d044d53-5c7d778998-sqrmn\" (UID: \"3c4995b5-ff59-4d08-bff2-8f49d69619a9\") " pod="openshift-ingress/payload-processing-e2e-mt-6d044d53-5c7d778998-sqrmn"
Jul 26 16:21:56.814404 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:56.814040    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95afe73b-8654-4e36-aa88-b28b9ca338f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6d044d53-644c44b476-hgv7f\" (UID: \"95afe73b-8654-4e36-aa88-b28b9ca338f2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6d044d53-644c44b476-hgv7f"
Jul 26 16:21:56.814404 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:56.814103    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c4995b5-ff59-4d08-bff2-8f49d69619a9-plugins-config-volume\") pod \"payload-processing-e2e-mt-6d044d53-5c7d778998-sqrmn\" (UID: \"3c4995b5-ff59-4d08-bff2-8f49d69619a9\") " pod="openshift-ingress/payload-processing-e2e-mt-6d044d53-5c7d778998-sqrmn"
Jul 26 16:21:56.814404 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:56.814176    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbrjp\" (UniqueName: \"kubernetes.io/projected/95afe73b-8654-4e36-aa88-b28b9ca338f2-kube-api-access-tbrjp\") pod \"payload-pre-processing-e2e-mt-6d044d53-644c44b476-hgv7f\" (UID: \"95afe73b-8654-4e36-aa88-b28b9ca338f2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6d044d53-644c44b476-hgv7f"
Jul 26 16:21:56.814731 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:56.814712    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95afe73b-8654-4e36-aa88-b28b9ca338f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6d044d53-644c44b476-hgv7f\" (UID: \"95afe73b-8654-4e36-aa88-b28b9ca338f2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6d044d53-644c44b476-hgv7f"
Jul 26 16:21:56.822261 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:56.822228    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbrjp\" (UniqueName: \"kubernetes.io/projected/95afe73b-8654-4e36-aa88-b28b9ca338f2-kube-api-access-tbrjp\") pod \"payload-pre-processing-e2e-mt-6d044d53-644c44b476-hgv7f\" (UID: \"95afe73b-8654-4e36-aa88-b28b9ca338f2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6d044d53-644c44b476-hgv7f"
Jul 26 16:21:56.888556 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:56.888513    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d044d53-786985db86-78b87"
Jul 26 16:21:56.915368 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:56.915296    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcb8v\" (UniqueName: \"kubernetes.io/projected/3c4995b5-ff59-4d08-bff2-8f49d69619a9-kube-api-access-pcb8v\") pod \"payload-processing-e2e-mt-6d044d53-5c7d778998-sqrmn\" (UID: \"3c4995b5-ff59-4d08-bff2-8f49d69619a9\") " pod="openshift-ingress/payload-processing-e2e-mt-6d044d53-5c7d778998-sqrmn"
Jul 26 16:21:56.915523 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:56.915384    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c4995b5-ff59-4d08-bff2-8f49d69619a9-plugins-config-volume\") pod \"payload-processing-e2e-mt-6d044d53-5c7d778998-sqrmn\" (UID: \"3c4995b5-ff59-4d08-bff2-8f49d69619a9\") " pod="openshift-ingress/payload-processing-e2e-mt-6d044d53-5c7d778998-sqrmn"
Jul 26 16:21:56.916071 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:56.916053    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c4995b5-ff59-4d08-bff2-8f49d69619a9-plugins-config-volume\") pod \"payload-processing-e2e-mt-6d044d53-5c7d778998-sqrmn\" (UID: \"3c4995b5-ff59-4d08-bff2-8f49d69619a9\") " pod="openshift-ingress/payload-processing-e2e-mt-6d044d53-5c7d778998-sqrmn"
Jul 26 16:21:56.923339 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:56.923317    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcb8v\" (UniqueName: \"kubernetes.io/projected/3c4995b5-ff59-4d08-bff2-8f49d69619a9-kube-api-access-pcb8v\") pod \"payload-processing-e2e-mt-6d044d53-5c7d778998-sqrmn\" (UID: \"3c4995b5-ff59-4d08-bff2-8f49d69619a9\") " pod="openshift-ingress/payload-processing-e2e-mt-6d044d53-5c7d778998-sqrmn"
Jul 26 16:21:56.963592 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:56.963567    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6d044d53-644c44b476-hgv7f"
Jul 26 16:21:56.982356 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:56.982282    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6d044d53-5c7d778998-sqrmn"
Jul 26 16:21:57.056724 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:57.055848    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6d044d53-786985db86-78b87"]
Jul 26 16:21:57.154842 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:57.154805    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6d044d53-644c44b476-hgv7f"]
Jul 26 16:21:57.158005 ip-10-0-133-204 kubenswrapper[2563]: W0726 16:21:57.157971    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95afe73b_8654_4e36_aa88_b28b9ca338f2.slice/crio-6c9ef562e1898869d03ef87d1d4a0dd246adfcadddd42636a7d14805e4d0158c WatchSource:0}: Error finding container 6c9ef562e1898869d03ef87d1d4a0dd246adfcadddd42636a7d14805e4d0158c: Status 404 returned error can't find the container with id 6c9ef562e1898869d03ef87d1d4a0dd246adfcadddd42636a7d14805e4d0158c
Jul 26 16:21:57.177553 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:57.177532    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6d044d53-5c7d778998-sqrmn"]
Jul 26 16:21:57.179481 ip-10-0-133-204 kubenswrapper[2563]: W0726 16:21:57.179452    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c4995b5_ff59_4d08_bff2_8f49d69619a9.slice/crio-99a9a88380f7572f706b1dff29280ccab3feac5a86e07bed393595d879338393 WatchSource:0}: Error finding container 99a9a88380f7572f706b1dff29280ccab3feac5a86e07bed393595d879338393: Status 404 returned error can't find the container with id 99a9a88380f7572f706b1dff29280ccab3feac5a86e07bed393595d879338393
Jul 26 16:21:57.629894 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:57.629864    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6d044d53-5c7d778998-sqrmn" event={"ID":"3c4995b5-ff59-4d08-bff2-8f49d69619a9","Type":"ContainerStarted","Data":"5b383590abdd7debdc828a9b8fed1bbac444b74c0558493a4ee717da63e9a7f4"}
Jul 26 16:21:57.630028 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:57.629925    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6d044d53-5c7d778998-sqrmn" event={"ID":"3c4995b5-ff59-4d08-bff2-8f49d69619a9","Type":"ContainerStarted","Data":"99a9a88380f7572f706b1dff29280ccab3feac5a86e07bed393595d879338393"}
Jul 26 16:21:57.630028 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:57.629954    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-6d044d53-5c7d778998-sqrmn"
Jul 26 16:21:57.631315 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:57.631289    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d044d53-786985db86-78b87" event={"ID":"dcd9528e-ba1a-4260-bba6-589e5377cba4","Type":"ContainerStarted","Data":"fd3ffd856640ecb86fff2f8418dae320f5d5e2f9098aaecd9c8450b29400549a"}
Jul 26 16:21:57.632982 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:57.632960    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6d044d53-644c44b476-hgv7f" event={"ID":"95afe73b-8654-4e36-aa88-b28b9ca338f2","Type":"ContainerStarted","Data":"037324a92d55cf9ac1fb760585a0ec1a273b4e8121e982f538c0b5027974ef7d"}
Jul 26 16:21:57.633101 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:57.632997    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6d044d53-644c44b476-hgv7f" event={"ID":"95afe73b-8654-4e36-aa88-b28b9ca338f2","Type":"ContainerStarted","Data":"6c9ef562e1898869d03ef87d1d4a0dd246adfcadddd42636a7d14805e4d0158c"}
Jul 26 16:21:57.633166 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:57.633114    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-6d044d53-644c44b476-hgv7f"
Jul 26 16:21:57.647815 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:57.647756    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-6d044d53-5c7d778998-sqrmn" podStartSLOduration=1.647739782 podStartE2EDuration="1.647739782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:21:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:21:57.646198345 +0000 UTC m=+1909.750424156" watchObservedRunningTime="2026-07-26 16:21:57.647739782 +0000 UTC m=+1909.751965591"
Jul 26 16:21:57.664020 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:57.663977    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-6d044d53-644c44b476-hgv7f" podStartSLOduration=1.6639648089999999 podStartE2EDuration="1.663964809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:21:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:21:57.662280104 +0000 UTC m=+1909.766505913" watchObservedRunningTime="2026-07-26 16:21:57.663964809 +0000 UTC m=+1909.768190653"
Jul 26 16:21:58.639736 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:58.639701    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d044d53-786985db86-78b87" event={"ID":"dcd9528e-ba1a-4260-bba6-589e5377cba4","Type":"ContainerStarted","Data":"8a13ee13269317b1e9147b47bc1f013530e4e2032fc77f0188796accc1176100"}
Jul 26 16:21:58.640209 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:58.640055    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d044d53-786985db86-78b87"
Jul 26 16:21:58.656779 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:21:58.656709    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d044d53-786985db86-78b87" podStartSLOduration=3.108104613 podStartE2EDuration="3.656694996s" totalImagesPullingTime="548.590383ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:21:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:21:58.655380558 +0000 UTC m=+1910.759606368" watchObservedRunningTime="2026-07-26 16:21:58.656694996 +0000 UTC m=+1910.760920805"
Jul 26 16:22:00.323688 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:00.323653    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:22:01.029806 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:01.029772    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:22:01.124972 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:01.124937    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:22:01.495143 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:01.495108    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:22:04.649701 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:04.649672    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d044d53-786985db86-78b87"
Jul 26 16:22:07.326886 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:07.326812    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:22:07.389378 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:07.389351    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:22:08.641821 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:08.641786    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-6d044d53-5c7d778998-sqrmn"
Jul 26 16:22:08.642229 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:08.641906    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-6d044d53-644c44b476-hgv7f"
Jul 26 16:22:18.077778 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:18.077728    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6d044d53-786985db86-78b87"]
Jul 26 16:22:18.080457 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:18.078458    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d044d53-786985db86-78b87" podUID="dcd9528e-ba1a-4260-bba6-589e5377cba4" containerName="maas-api" containerID="cri-o://8a13ee13269317b1e9147b47bc1f013530e4e2032fc77f0188796accc1176100" gracePeriod=30
Jul 26 16:22:18.108279 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:18.108237    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:22:18.127570 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:18.127536    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6d044d53-5c7d778998-sqrmn"]
Jul 26 16:22:18.127790 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:18.127750    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-6d044d53-5c7d778998-sqrmn" podUID="3c4995b5-ff59-4d08-bff2-8f49d69619a9" containerName="payload-processing" containerID="cri-o://5b383590abdd7debdc828a9b8fed1bbac444b74c0558493a4ee717da63e9a7f4" gracePeriod=30
Jul 26 16:22:18.135266 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:18.135235    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6d044d53-644c44b476-hgv7f"]
Jul 26 16:22:18.135599 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:18.135555    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-6d044d53-644c44b476-hgv7f" podUID="95afe73b-8654-4e36-aa88-b28b9ca338f2" containerName="payload-pre-processing" containerID="cri-o://037324a92d55cf9ac1fb760585a0ec1a273b4e8121e982f538c0b5027974ef7d" gracePeriod=30
Jul 26 16:22:18.165944 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:18.165915    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:22:18.227469 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:18.227438    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:22:18.400961 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:18.400931    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d044d53-786985db86-78b87"
Jul 26 16:22:18.413982 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:18.413959    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6d044d53-5c7d778998-sqrmn"
Jul 26 16:22:18.417845 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:18.417799    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-45h7g\" (UniqueName: \"kubernetes.io/projected/dcd9528e-ba1a-4260-bba6-589e5377cba4-kube-api-access-45h7g\") pod \"dcd9528e-ba1a-4260-bba6-589e5377cba4\" (UID: \"dcd9528e-ba1a-4260-bba6-589e5377cba4\") "
Jul 26 16:22:18.419888 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:18.419528    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcd9528e-ba1a-4260-bba6-589e5377cba4-openshift-service-ca\") pod \"dcd9528e-ba1a-4260-bba6-589e5377cba4\" (UID: \"dcd9528e-ba1a-4260-bba6-589e5377cba4\") "
Jul 26 16:22:18.419888 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:18.419582    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dcd9528e-ba1a-4260-bba6-589e5377cba4-maas-api-tls\") pod \"dcd9528e-ba1a-4260-bba6-589e5377cba4\" (UID: \"dcd9528e-ba1a-4260-bba6-589e5377cba4\") "
Jul 26 16:22:18.420062 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:18.419930    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcd9528e-ba1a-4260-bba6-589e5377cba4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "dcd9528e-ba1a-4260-bba6-589e5377cba4" (UID: "dcd9528e-ba1a-4260-bba6-589e5377cba4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 26 16:22:18.421068 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:18.420992    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcd9528e-ba1a-4260-bba6-589e5377cba4-kube-api-access-45h7g" (OuterVolumeSpecName: "kube-api-access-45h7g") pod "dcd9528e-ba1a-4260-bba6-589e5377cba4" (UID: "dcd9528e-ba1a-4260-bba6-589e5377cba4"). InnerVolumeSpecName "kube-api-access-45h7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 26 16:22:18.422350 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:18.422323    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcd9528e-ba1a-4260-bba6-589e5377cba4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "dcd9528e-ba1a-4260-bba6-589e5377cba4" (UID: "dcd9528e-ba1a-4260-bba6-589e5377cba4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 26 16:22:18.435418 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:18.435396    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6d044d53-644c44b476-hgv7f"
Jul 26 16:22:18.520933 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:18.520892    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pcb8v\" (UniqueName: \"kubernetes.io/projected/3c4995b5-ff59-4d08-bff2-8f49d69619a9-kube-api-access-pcb8v\") pod \"3c4995b5-ff59-4d08-bff2-8f49d69619a9\" (UID: \"3c4995b5-ff59-4d08-bff2-8f49d69619a9\") "
Jul 26 16:22:18.521113 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:18.520964    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c4995b5-ff59-4d08-bff2-8f49d69619a9-plugins-config-volume\") pod \"3c4995b5-ff59-4d08-bff2-8f49d69619a9\" (UID: \"3c4995b5-ff59-4d08-bff2-8f49d69619a9\") "
Jul 26 16:22:18.521113 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:18.521031    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tbrjp\" (UniqueName: \"kubernetes.io/projected/95afe73b-8654-4e36-aa88-b28b9ca338f2-kube-api-access-tbrjp\") pod \"95afe73b-8654-4e36-aa88-b28b9ca338f2\" (UID: \"95afe73b-8654-4e36-aa88-b28b9ca338f2\") "
Jul 26 16:22:18.521113 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:18.521056    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95afe73b-8654-4e36-aa88-b28b9ca338f2-plugins-config-volume\") pod \"95afe73b-8654-4e36-aa88-b28b9ca338f2\" (UID: \"95afe73b-8654-4e36-aa88-b28b9ca338f2\") "
Jul 26 16:22:18.521377 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:18.521355    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45h7g\" (UniqueName: \"kubernetes.io/projected/dcd9528e-ba1a-4260-bba6-589e5377cba4-kube-api-access-45h7g\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:22:18.521444 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:18.521382    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcd9528e-ba1a-4260-bba6-589e5377cba4-openshift-service-ca\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:22:18.521444 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:18.521396    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dcd9528e-ba1a-4260-bba6-589e5377cba4-maas-api-tls\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:22:18.521444 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:18.521378    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c4995b5-ff59-4d08-bff2-8f49d69619a9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3c4995b5-ff59-4d08-bff2-8f49d69619a9" (UID: "3c4995b5-ff59-4d08-bff2-8f49d69619a9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 26 16:22:18.521444 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:18.521423    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95afe73b-8654-4e36-aa88-b28b9ca338f2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "95afe73b-8654-4e36-aa88-b28b9ca338f2" (UID: "95afe73b-8654-4e36-aa88-b28b9ca338f2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 26 16:22:18.523161 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:18.523141    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95afe73b-8654-4e36-aa88-b28b9ca338f2-kube-api-access-tbrjp" (OuterVolumeSpecName: "kube-api-access-tbrjp") pod "95afe73b-8654-4e36-aa88-b28b9ca338f2" (UID: "95afe73b-8654-4e36-aa88-b28b9ca338f2"). InnerVolumeSpecName "kube-api-access-tbrjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 26 16:22:18.523219 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:18.523174    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c4995b5-ff59-4d08-bff2-8f49d69619a9-kube-api-access-pcb8v" (OuterVolumeSpecName: "kube-api-access-pcb8v") pod "3c4995b5-ff59-4d08-bff2-8f49d69619a9" (UID: "3c4995b5-ff59-4d08-bff2-8f49d69619a9"). InnerVolumeSpecName "kube-api-access-pcb8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 26 16:22:18.622373 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:18.622288    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcb8v\" (UniqueName: \"kubernetes.io/projected/3c4995b5-ff59-4d08-bff2-8f49d69619a9-kube-api-access-pcb8v\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:22:18.622373 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:18.622324    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c4995b5-ff59-4d08-bff2-8f49d69619a9-plugins-config-volume\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:22:18.622373 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:18.622367    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbrjp\" (UniqueName: \"kubernetes.io/projected/95afe73b-8654-4e36-aa88-b28b9ca338f2-kube-api-access-tbrjp\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:22:18.622599 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:18.622382    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95afe73b-8654-4e36-aa88-b28b9ca338f2-plugins-config-volume\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:22:18.741091 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:18.741058    2563 generic.go:356] "Generic (PLEG): container finished" podID="dcd9528e-ba1a-4260-bba6-589e5377cba4" containerID="8a13ee13269317b1e9147b47bc1f013530e4e2032fc77f0188796accc1176100" exitCode=0
Jul 26 16:22:18.741288 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:18.741108    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d044d53-786985db86-78b87"
Jul 26 16:22:18.741288 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:18.741144    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d044d53-786985db86-78b87" event={"ID":"dcd9528e-ba1a-4260-bba6-589e5377cba4","Type":"ContainerDied","Data":"8a13ee13269317b1e9147b47bc1f013530e4e2032fc77f0188796accc1176100"}
Jul 26 16:22:18.741288 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:18.741200    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d044d53-786985db86-78b87" event={"ID":"dcd9528e-ba1a-4260-bba6-589e5377cba4","Type":"ContainerDied","Data":"fd3ffd856640ecb86fff2f8418dae320f5d5e2f9098aaecd9c8450b29400549a"}
Jul 26 16:22:18.741288 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:18.741229    2563 scope.go:117] "RemoveContainer" containerID="8a13ee13269317b1e9147b47bc1f013530e4e2032fc77f0188796accc1176100"
Jul 26 16:22:18.742461 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:18.742432    2563 generic.go:356] "Generic (PLEG): container finished" podID="95afe73b-8654-4e36-aa88-b28b9ca338f2" containerID="037324a92d55cf9ac1fb760585a0ec1a273b4e8121e982f538c0b5027974ef7d" exitCode=0
Jul 26 16:22:18.742575 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:18.742530    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6d044d53-644c44b476-hgv7f" event={"ID":"95afe73b-8654-4e36-aa88-b28b9ca338f2","Type":"ContainerDied","Data":"037324a92d55cf9ac1fb760585a0ec1a273b4e8121e982f538c0b5027974ef7d"}
Jul 26 16:22:18.742707 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:18.742576    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6d044d53-644c44b476-hgv7f" event={"ID":"95afe73b-8654-4e36-aa88-b28b9ca338f2","Type":"ContainerDied","Data":"6c9ef562e1898869d03ef87d1d4a0dd246adfcadddd42636a7d14805e4d0158c"}
Jul 26 16:22:18.742707 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:18.742589    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6d044d53-644c44b476-hgv7f"
Jul 26 16:22:18.752639 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:18.752600    2563 generic.go:356] "Generic (PLEG): container finished" podID="3c4995b5-ff59-4d08-bff2-8f49d69619a9" containerID="5b383590abdd7debdc828a9b8fed1bbac444b74c0558493a4ee717da63e9a7f4" exitCode=0
Jul 26 16:22:18.752808 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:18.752658    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6d044d53-5c7d778998-sqrmn"
Jul 26 16:22:18.752808 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:18.752688    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6d044d53-5c7d778998-sqrmn" event={"ID":"3c4995b5-ff59-4d08-bff2-8f49d69619a9","Type":"ContainerDied","Data":"5b383590abdd7debdc828a9b8fed1bbac444b74c0558493a4ee717da63e9a7f4"}
Jul 26 16:22:18.752808 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:18.752796    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6d044d53-5c7d778998-sqrmn" event={"ID":"3c4995b5-ff59-4d08-bff2-8f49d69619a9","Type":"ContainerDied","Data":"99a9a88380f7572f706b1dff29280ccab3feac5a86e07bed393595d879338393"}
Jul 26 16:22:18.758103 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:18.758025    2563 scope.go:117] "RemoveContainer" containerID="8a13ee13269317b1e9147b47bc1f013530e4e2032fc77f0188796accc1176100"
Jul 26 16:22:18.758355 ip-10-0-133-204 kubenswrapper[2563]: E0726 16:22:18.758329    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a13ee13269317b1e9147b47bc1f013530e4e2032fc77f0188796accc1176100\": container with ID starting with 8a13ee13269317b1e9147b47bc1f013530e4e2032fc77f0188796accc1176100 not found: ID does not exist" containerID="8a13ee13269317b1e9147b47bc1f013530e4e2032fc77f0188796accc1176100"
Jul 26 16:22:18.758440 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:18.758367    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a13ee13269317b1e9147b47bc1f013530e4e2032fc77f0188796accc1176100"} err="failed to get container status \"8a13ee13269317b1e9147b47bc1f013530e4e2032fc77f0188796accc1176100\": rpc error: code = NotFound desc = could not find container \"8a13ee13269317b1e9147b47bc1f013530e4e2032fc77f0188796accc1176100\": container with ID starting with 8a13ee13269317b1e9147b47bc1f013530e4e2032fc77f0188796accc1176100 not found: ID does not exist"
Jul 26 16:22:18.758440 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:18.758389    2563 scope.go:117] "RemoveContainer" containerID="037324a92d55cf9ac1fb760585a0ec1a273b4e8121e982f538c0b5027974ef7d"
Jul 26 16:22:18.767846 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:18.767828    2563 scope.go:117] "RemoveContainer" containerID="037324a92d55cf9ac1fb760585a0ec1a273b4e8121e982f538c0b5027974ef7d"
Jul 26 16:22:18.768106 ip-10-0-133-204 kubenswrapper[2563]: E0726 16:22:18.768088    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"037324a92d55cf9ac1fb760585a0ec1a273b4e8121e982f538c0b5027974ef7d\": container with ID starting with 037324a92d55cf9ac1fb760585a0ec1a273b4e8121e982f538c0b5027974ef7d not found: ID does not exist" containerID="037324a92d55cf9ac1fb760585a0ec1a273b4e8121e982f538c0b5027974ef7d"
Jul 26 16:22:18.768161 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:18.768115    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"037324a92d55cf9ac1fb760585a0ec1a273b4e8121e982f538c0b5027974ef7d"} err="failed to get container status \"037324a92d55cf9ac1fb760585a0ec1a273b4e8121e982f538c0b5027974ef7d\": rpc error: code = NotFound desc = could not find container \"037324a92d55cf9ac1fb760585a0ec1a273b4e8121e982f538c0b5027974ef7d\": container with ID starting with 037324a92d55cf9ac1fb760585a0ec1a273b4e8121e982f538c0b5027974ef7d not found: ID does not exist"
Jul 26 16:22:18.768161 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:18.768132    2563 scope.go:117] "RemoveContainer" containerID="5b383590abdd7debdc828a9b8fed1bbac444b74c0558493a4ee717da63e9a7f4"
Jul 26 16:22:18.768512 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:18.768494    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6d044d53-786985db86-78b87"]
Jul 26 16:22:18.773301 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:18.773282    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6d044d53-786985db86-78b87"]
Jul 26 16:22:18.776173 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:18.776152    2563 scope.go:117] "RemoveContainer" containerID="5b383590abdd7debdc828a9b8fed1bbac444b74c0558493a4ee717da63e9a7f4"
Jul 26 16:22:18.776394 ip-10-0-133-204 kubenswrapper[2563]: E0726 16:22:18.776378    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b383590abdd7debdc828a9b8fed1bbac444b74c0558493a4ee717da63e9a7f4\": container with ID starting with 5b383590abdd7debdc828a9b8fed1bbac444b74c0558493a4ee717da63e9a7f4 not found: ID does not exist" containerID="5b383590abdd7debdc828a9b8fed1bbac444b74c0558493a4ee717da63e9a7f4"
Jul 26 16:22:18.776446 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:18.776401    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b383590abdd7debdc828a9b8fed1bbac444b74c0558493a4ee717da63e9a7f4"} err="failed to get container status \"5b383590abdd7debdc828a9b8fed1bbac444b74c0558493a4ee717da63e9a7f4\": rpc error: code = NotFound desc = could not find container \"5b383590abdd7debdc828a9b8fed1bbac444b74c0558493a4ee717da63e9a7f4\": container with ID starting with 5b383590abdd7debdc828a9b8fed1bbac444b74c0558493a4ee717da63e9a7f4 not found: ID does not exist"
Jul 26 16:22:18.782517 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:18.782494    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6d044d53-5c7d778998-sqrmn"]
Jul 26 16:22:18.787494 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:18.787474    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6d044d53-5c7d778998-sqrmn"]
Jul 26 16:22:18.797056 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:18.797034    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6d044d53-644c44b476-hgv7f"]
Jul 26 16:22:18.802855 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:18.802833    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6d044d53-644c44b476-hgv7f"]
Jul 26 16:22:20.373416 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:20.373385    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c4995b5-ff59-4d08-bff2-8f49d69619a9" path="/var/lib/kubelet/pods/3c4995b5-ff59-4d08-bff2-8f49d69619a9/volumes"
Jul 26 16:22:20.373832 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:20.373724    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95afe73b-8654-4e36-aa88-b28b9ca338f2" path="/var/lib/kubelet/pods/95afe73b-8654-4e36-aa88-b28b9ca338f2/volumes"
Jul 26 16:22:20.374104 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:20.374090    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dcd9528e-ba1a-4260-bba6-589e5377cba4" path="/var/lib/kubelet/pods/dcd9528e-ba1a-4260-bba6-589e5377cba4/volumes"
Jul 26 16:22:22.754242 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:22.754213    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:22:23.882229 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:23.882199    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:22:29.128631 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:29.128597    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:22:29.193140 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:29.193104    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:22:29.424385 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:29.424293    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:22:29.666282 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:29.666253    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:22:29.801294 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:29.801239    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:22:29.896466 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:29.896407    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:22:30.013382 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:30.013324    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:22:35.579200 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:35.579167    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:22:35.693423 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:35.693393    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:22:35.780078 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:35.780044    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:22:35.949619 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:35.949520    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:22:41.582633 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:41.582596    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:22:41.685667 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:41.685633    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:22:48.534162 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:48.534127    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:22:48.896234 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:48.896083    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:22:49.297213 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:49.297174    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:22:51.908250 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:51.908206    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:22:52.018669 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:52.018637    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-631b3d03-67f5d946d6-f5t7n"]
Jul 26 16:22:52.019257 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:52.019238    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcd9528e-ba1a-4260-bba6-589e5377cba4" containerName="maas-api"
Jul 26 16:22:52.019257 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:52.019258    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcd9528e-ba1a-4260-bba6-589e5377cba4" containerName="maas-api"
Jul 26 16:22:52.019378 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:52.019291    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95afe73b-8654-4e36-aa88-b28b9ca338f2" containerName="payload-pre-processing"
Jul 26 16:22:52.019378 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:52.019299    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="95afe73b-8654-4e36-aa88-b28b9ca338f2" containerName="payload-pre-processing"
Jul 26 16:22:52.019378 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:52.019318    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c4995b5-ff59-4d08-bff2-8f49d69619a9" containerName="payload-processing"
Jul 26 16:22:52.019378 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:52.019327    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c4995b5-ff59-4d08-bff2-8f49d69619a9" containerName="payload-processing"
Jul 26 16:22:52.019527 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:52.019403    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcd9528e-ba1a-4260-bba6-589e5377cba4" containerName="maas-api"
Jul 26 16:22:52.019527 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:52.019417    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="95afe73b-8654-4e36-aa88-b28b9ca338f2" containerName="payload-pre-processing"
Jul 26 16:22:52.019527 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:52.019427    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c4995b5-ff59-4d08-bff2-8f49d69619a9" containerName="payload-processing"
Jul 26 16:22:52.022732 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:52.022713    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-631b3d03-67f5d946d6-f5t7n"
Jul 26 16:22:52.026449 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:52.026427    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-631b3d03-serving-cert\""
Jul 26 16:22:52.035627 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:52.035600    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:22:52.036211 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:52.036191    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-631b3d03-67f5d946d6-f5t7n"]
Jul 26 16:22:52.132026 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:52.131986    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:22:52.138664 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:52.138624    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49ccf\" (UniqueName: \"kubernetes.io/projected/a229b032-a66e-4eda-9b0a-587219ae1c2b-kube-api-access-49ccf\") pod \"maas-api-e2e-mt-631b3d03-67f5d946d6-f5t7n\" (UID: \"a229b032-a66e-4eda-9b0a-587219ae1c2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-631b3d03-67f5d946d6-f5t7n"
Jul 26 16:22:52.138871 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:52.138701    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a229b032-a66e-4eda-9b0a-587219ae1c2b-maas-api-tls\") pod \"maas-api-e2e-mt-631b3d03-67f5d946d6-f5t7n\" (UID: \"a229b032-a66e-4eda-9b0a-587219ae1c2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-631b3d03-67f5d946d6-f5t7n"
Jul 26 16:22:52.138937 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:52.138870    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a229b032-a66e-4eda-9b0a-587219ae1c2b-openshift-service-ca\") pod \"maas-api-e2e-mt-631b3d03-67f5d946d6-f5t7n\" (UID: \"a229b032-a66e-4eda-9b0a-587219ae1c2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-631b3d03-67f5d946d6-f5t7n"
Jul 26 16:22:52.240107 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:52.240074    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a229b032-a66e-4eda-9b0a-587219ae1c2b-maas-api-tls\") pod \"maas-api-e2e-mt-631b3d03-67f5d946d6-f5t7n\" (UID: \"a229b032-a66e-4eda-9b0a-587219ae1c2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-631b3d03-67f5d946d6-f5t7n"
Jul 26 16:22:52.240286 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:52.240204    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a229b032-a66e-4eda-9b0a-587219ae1c2b-openshift-service-ca\") pod \"maas-api-e2e-mt-631b3d03-67f5d946d6-f5t7n\" (UID: \"a229b032-a66e-4eda-9b0a-587219ae1c2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-631b3d03-67f5d946d6-f5t7n"
Jul 26 16:22:52.240286 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:52.240247    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49ccf\" (UniqueName: \"kubernetes.io/projected/a229b032-a66e-4eda-9b0a-587219ae1c2b-kube-api-access-49ccf\") pod \"maas-api-e2e-mt-631b3d03-67f5d946d6-f5t7n\" (UID: \"a229b032-a66e-4eda-9b0a-587219ae1c2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-631b3d03-67f5d946d6-f5t7n"
Jul 26 16:22:52.240899 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:52.240875    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a229b032-a66e-4eda-9b0a-587219ae1c2b-openshift-service-ca\") pod \"maas-api-e2e-mt-631b3d03-67f5d946d6-f5t7n\" (UID: \"a229b032-a66e-4eda-9b0a-587219ae1c2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-631b3d03-67f5d946d6-f5t7n"
Jul 26 16:22:52.242733 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:52.242709    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a229b032-a66e-4eda-9b0a-587219ae1c2b-maas-api-tls\") pod \"maas-api-e2e-mt-631b3d03-67f5d946d6-f5t7n\" (UID: \"a229b032-a66e-4eda-9b0a-587219ae1c2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-631b3d03-67f5d946d6-f5t7n"
Jul 26 16:22:52.248231 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:52.248211    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49ccf\" (UniqueName: \"kubernetes.io/projected/a229b032-a66e-4eda-9b0a-587219ae1c2b-kube-api-access-49ccf\") pod \"maas-api-e2e-mt-631b3d03-67f5d946d6-f5t7n\" (UID: \"a229b032-a66e-4eda-9b0a-587219ae1c2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-631b3d03-67f5d946d6-f5t7n"
Jul 26 16:22:52.263749 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:52.263723    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:22:52.332628 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:52.332587    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-631b3d03-67f5d946d6-f5t7n"
Jul 26 16:22:52.480269 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:52.480242    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-631b3d03-67f5d946d6-f5t7n"]
Jul 26 16:22:52.482619 ip-10-0-133-204 kubenswrapper[2563]: W0726 16:22:52.482593    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda229b032_a66e_4eda_9b0a_587219ae1c2b.slice/crio-45e23d4c23ea65221b2d015f65e47a6c0ce6a046765d956a7c6b1205ab79b1fb WatchSource:0}: Error finding container 45e23d4c23ea65221b2d015f65e47a6c0ce6a046765d956a7c6b1205ab79b1fb: Status 404 returned error can't find the container with id 45e23d4c23ea65221b2d015f65e47a6c0ce6a046765d956a7c6b1205ab79b1fb
Jul 26 16:22:52.484390 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:52.484369    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 26 16:22:52.591607 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:52.591500    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:22:52.688466 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:52.688426    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-631b3d03-6454c8fdd4-dsvlp"]
Jul 26 16:22:52.690035 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:52.689940    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:22:52.696254 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:52.696234    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-631b3d03-6454c8fdd4-dsvlp"
Jul 26 16:22:52.699774 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:52.699737    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-631b3d03\""
Jul 26 16:22:52.703458 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:52.703424    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-631b3d03-6454c8fdd4-dsvlp"]
Jul 26 16:22:52.719062 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:52.719027    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-631b3d03-757985dbbb-mv4l6"]
Jul 26 16:22:52.722988 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:52.722964    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-631b3d03-757985dbbb-mv4l6"
Jul 26 16:22:52.725927 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:52.725905    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-631b3d03-dockercfg-wcjvs\""
Jul 26 16:22:52.735947 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:52.735919    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-631b3d03-757985dbbb-mv4l6"]
Jul 26 16:22:52.845445 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:52.845382    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53bf2ad2-8173-49f4-9056-0f575eb57695-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-631b3d03-6454c8fdd4-dsvlp\" (UID: \"53bf2ad2-8173-49f4-9056-0f575eb57695\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-631b3d03-6454c8fdd4-dsvlp"
Jul 26 16:22:52.845564 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:52.845505    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6829\" (UniqueName: \"kubernetes.io/projected/53bf2ad2-8173-49f4-9056-0f575eb57695-kube-api-access-k6829\") pod \"payload-pre-processing-e2e-mt-631b3d03-6454c8fdd4-dsvlp\" (UID: \"53bf2ad2-8173-49f4-9056-0f575eb57695\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-631b3d03-6454c8fdd4-dsvlp"
Jul 26 16:22:52.845564 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:52.845543    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9514fe11-e115-416e-bebd-7cea1bc5af4a-plugins-config-volume\") pod \"payload-processing-e2e-mt-631b3d03-757985dbbb-mv4l6\" (UID: \"9514fe11-e115-416e-bebd-7cea1bc5af4a\") " pod="openshift-ingress/payload-processing-e2e-mt-631b3d03-757985dbbb-mv4l6"
Jul 26 16:22:52.845679 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:52.845590    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xdbj\" (UniqueName: \"kubernetes.io/projected/9514fe11-e115-416e-bebd-7cea1bc5af4a-kube-api-access-4xdbj\") pod \"payload-processing-e2e-mt-631b3d03-757985dbbb-mv4l6\" (UID: \"9514fe11-e115-416e-bebd-7cea1bc5af4a\") " pod="openshift-ingress/payload-processing-e2e-mt-631b3d03-757985dbbb-mv4l6"
Jul 26 16:22:52.921444 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:52.921401    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-631b3d03-67f5d946d6-f5t7n" event={"ID":"a229b032-a66e-4eda-9b0a-587219ae1c2b","Type":"ContainerStarted","Data":"4a29898cd6283bc89cff841facdc106053a42695001dcd0cf91741b0522c4ebd"}
Jul 26 16:22:52.921855 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:52.921473    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-631b3d03-67f5d946d6-f5t7n"
Jul 26 16:22:52.921855 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:52.921498    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-631b3d03-67f5d946d6-f5t7n" event={"ID":"a229b032-a66e-4eda-9b0a-587219ae1c2b","Type":"ContainerStarted","Data":"45e23d4c23ea65221b2d015f65e47a6c0ce6a046765d956a7c6b1205ab79b1fb"}
Jul 26 16:22:52.945721 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:52.945669    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-631b3d03-67f5d946d6-f5t7n" podStartSLOduration=1.59460566 podStartE2EDuration="1.945650898s" totalImagesPullingTime="351.045238ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:22:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:22:52.944171742 +0000 UTC m=+1965.048397551" watchObservedRunningTime="2026-07-26 16:22:52.945650898 +0000 UTC m=+1965.049876726"
Jul 26 16:22:52.947201 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:52.947174    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6829\" (UniqueName: \"kubernetes.io/projected/53bf2ad2-8173-49f4-9056-0f575eb57695-kube-api-access-k6829\") pod \"payload-pre-processing-e2e-mt-631b3d03-6454c8fdd4-dsvlp\" (UID: \"53bf2ad2-8173-49f4-9056-0f575eb57695\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-631b3d03-6454c8fdd4-dsvlp"
Jul 26 16:22:52.947283 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:52.947223    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9514fe11-e115-416e-bebd-7cea1bc5af4a-plugins-config-volume\") pod \"payload-processing-e2e-mt-631b3d03-757985dbbb-mv4l6\" (UID: \"9514fe11-e115-416e-bebd-7cea1bc5af4a\") " pod="openshift-ingress/payload-processing-e2e-mt-631b3d03-757985dbbb-mv4l6"
Jul 26 16:22:52.947341 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:52.947322    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xdbj\" (UniqueName: \"kubernetes.io/projected/9514fe11-e115-416e-bebd-7cea1bc5af4a-kube-api-access-4xdbj\") pod \"payload-processing-e2e-mt-631b3d03-757985dbbb-mv4l6\" (UID: \"9514fe11-e115-416e-bebd-7cea1bc5af4a\") " pod="openshift-ingress/payload-processing-e2e-mt-631b3d03-757985dbbb-mv4l6"
Jul 26 16:22:52.947397 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:52.947383    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53bf2ad2-8173-49f4-9056-0f575eb57695-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-631b3d03-6454c8fdd4-dsvlp\" (UID: \"53bf2ad2-8173-49f4-9056-0f575eb57695\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-631b3d03-6454c8fdd4-dsvlp"
Jul 26 16:22:52.947901 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:52.947883    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9514fe11-e115-416e-bebd-7cea1bc5af4a-plugins-config-volume\") pod \"payload-processing-e2e-mt-631b3d03-757985dbbb-mv4l6\" (UID: \"9514fe11-e115-416e-bebd-7cea1bc5af4a\") " pod="openshift-ingress/payload-processing-e2e-mt-631b3d03-757985dbbb-mv4l6"
Jul 26 16:22:52.948070 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:52.948054    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53bf2ad2-8173-49f4-9056-0f575eb57695-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-631b3d03-6454c8fdd4-dsvlp\" (UID: \"53bf2ad2-8173-49f4-9056-0f575eb57695\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-631b3d03-6454c8fdd4-dsvlp"
Jul 26 16:22:52.960422 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:52.960390    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xdbj\" (UniqueName: \"kubernetes.io/projected/9514fe11-e115-416e-bebd-7cea1bc5af4a-kube-api-access-4xdbj\") pod \"payload-processing-e2e-mt-631b3d03-757985dbbb-mv4l6\" (UID: \"9514fe11-e115-416e-bebd-7cea1bc5af4a\") " pod="openshift-ingress/payload-processing-e2e-mt-631b3d03-757985dbbb-mv4l6"
Jul 26 16:22:52.960554 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:52.960425    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6829\" (UniqueName: \"kubernetes.io/projected/53bf2ad2-8173-49f4-9056-0f575eb57695-kube-api-access-k6829\") pod \"payload-pre-processing-e2e-mt-631b3d03-6454c8fdd4-dsvlp\" (UID: \"53bf2ad2-8173-49f4-9056-0f575eb57695\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-631b3d03-6454c8fdd4-dsvlp"
Jul 26 16:22:53.009187 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:53.009158    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-631b3d03-6454c8fdd4-dsvlp"
Jul 26 16:22:53.036299 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:53.036261    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-631b3d03-757985dbbb-mv4l6"
Jul 26 16:22:53.168325 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:53.167975    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-631b3d03-6454c8fdd4-dsvlp"]
Jul 26 16:22:53.171491 ip-10-0-133-204 kubenswrapper[2563]: W0726 16:22:53.171460    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53bf2ad2_8173_49f4_9056_0f575eb57695.slice/crio-425288cde71ebc981da6f92a9ce8fc2f8f56251bb0079a2d39e7160fbc5ba7c8 WatchSource:0}: Error finding container 425288cde71ebc981da6f92a9ce8fc2f8f56251bb0079a2d39e7160fbc5ba7c8: Status 404 returned error can't find the container with id 425288cde71ebc981da6f92a9ce8fc2f8f56251bb0079a2d39e7160fbc5ba7c8
Jul 26 16:22:53.411296 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:53.411264    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-631b3d03-757985dbbb-mv4l6"]
Jul 26 16:22:53.413661 ip-10-0-133-204 kubenswrapper[2563]: W0726 16:22:53.413633    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9514fe11_e115_416e_bebd_7cea1bc5af4a.slice/crio-2deb882cf7f3b332bcfca780f8fa55fdd8adbfcfd73e170cb4e7222384c6e49c WatchSource:0}: Error finding container 2deb882cf7f3b332bcfca780f8fa55fdd8adbfcfd73e170cb4e7222384c6e49c: Status 404 returned error can't find the container with id 2deb882cf7f3b332bcfca780f8fa55fdd8adbfcfd73e170cb4e7222384c6e49c
Jul 26 16:22:53.721218 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:53.721091    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:22:53.831383 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:53.831348    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:22:53.929835 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:53.929787    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-631b3d03-757985dbbb-mv4l6" event={"ID":"9514fe11-e115-416e-bebd-7cea1bc5af4a","Type":"ContainerStarted","Data":"14f17233cdc3a64367b8739f87cf51ca1b555c2dbce76fd1296f45829fa83397"}
Jul 26 16:22:53.930340 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:53.930017    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-631b3d03-757985dbbb-mv4l6"
Jul 26 16:22:53.930340 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:53.930061    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-631b3d03-757985dbbb-mv4l6" event={"ID":"9514fe11-e115-416e-bebd-7cea1bc5af4a","Type":"ContainerStarted","Data":"2deb882cf7f3b332bcfca780f8fa55fdd8adbfcfd73e170cb4e7222384c6e49c"}
Jul 26 16:22:53.931460 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:53.931429    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-631b3d03-6454c8fdd4-dsvlp" event={"ID":"53bf2ad2-8173-49f4-9056-0f575eb57695","Type":"ContainerStarted","Data":"1b2f1a41c29ffc851a090c089c40d5080e798e89bbac305f13d3a0d7b2876b55"}
Jul 26 16:22:53.931567 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:53.931493    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-631b3d03-6454c8fdd4-dsvlp" event={"ID":"53bf2ad2-8173-49f4-9056-0f575eb57695","Type":"ContainerStarted","Data":"425288cde71ebc981da6f92a9ce8fc2f8f56251bb0079a2d39e7160fbc5ba7c8"}
Jul 26 16:22:53.948741 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:53.948686    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-631b3d03-757985dbbb-mv4l6" podStartSLOduration=1.948671349 podStartE2EDuration="1.948671349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:22:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:22:53.946433275 +0000 UTC m=+1966.050659084" watchObservedRunningTime="2026-07-26 16:22:53.948671349 +0000 UTC m=+1966.052897157"
Jul 26 16:22:53.962577 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:53.962524    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-631b3d03-6454c8fdd4-dsvlp" podStartSLOduration=1.962507151 podStartE2EDuration="1.962507151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:22:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:22:53.962038703 +0000 UTC m=+1966.066264512" watchObservedRunningTime="2026-07-26 16:22:53.962507151 +0000 UTC m=+1966.066732961"
Jul 26 16:22:54.936341 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:54.936311    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-631b3d03-6454c8fdd4-dsvlp"
Jul 26 16:22:55.497087 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:55.497057    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:22:56.375813 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:56.375777    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:22:58.937545 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:22:58.937520    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-631b3d03-67f5d946d6-f5t7n"
Jul 26 16:23:00.379647 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:23:00.379577    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:23:04.938887 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:23:04.938860    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-631b3d03-757985dbbb-mv4l6"
Jul 26 16:23:05.944203 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:23:05.944176    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-631b3d03-6454c8fdd4-dsvlp"
Jul 26 16:23:36.330925 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:23:36.330884    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-631b3d03-67f5d946d6-f5t7n"]
Jul 26 16:23:36.331477 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:23:36.331175    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-631b3d03-67f5d946d6-f5t7n" podUID="a229b032-a66e-4eda-9b0a-587219ae1c2b" containerName="maas-api" containerID="cri-o://4a29898cd6283bc89cff841facdc106053a42695001dcd0cf91741b0522c4ebd" gracePeriod=30
Jul 26 16:23:36.364224 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:23:36.364191    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:23:36.394168 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:23:36.394135    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-631b3d03-757985dbbb-mv4l6"]
Jul 26 16:23:36.394430 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:23:36.394382    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-631b3d03-757985dbbb-mv4l6" podUID="9514fe11-e115-416e-bebd-7cea1bc5af4a" containerName="payload-processing" containerID="cri-o://14f17233cdc3a64367b8739f87cf51ca1b555c2dbce76fd1296f45829fa83397" gracePeriod=30
Jul 26 16:23:36.401449 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:23:36.401417    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-631b3d03-6454c8fdd4-dsvlp"]
Jul 26 16:23:36.401711 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:23:36.401641    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-631b3d03-6454c8fdd4-dsvlp" podUID="53bf2ad2-8173-49f4-9056-0f575eb57695" containerName="payload-pre-processing" containerID="cri-o://1b2f1a41c29ffc851a090c089c40d5080e798e89bbac305f13d3a0d7b2876b55" gracePeriod=30
Jul 26 16:23:36.460727 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:23:36.460692    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:23:36.562175 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:23:36.562105    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:23:36.666556 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:23:36.666533    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-631b3d03-67f5d946d6-f5t7n"
Jul 26 16:23:36.670461 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:23:36.670436    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-631b3d03-757985dbbb-mv4l6"
Jul 26 16:23:36.707549 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:23:36.707521    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-631b3d03-6454c8fdd4-dsvlp"
Jul 26 16:23:36.767492 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:23:36.767465    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a229b032-a66e-4eda-9b0a-587219ae1c2b-openshift-service-ca\") pod \"a229b032-a66e-4eda-9b0a-587219ae1c2b\" (UID: \"a229b032-a66e-4eda-9b0a-587219ae1c2b\") "
Jul 26 16:23:36.767703 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:23:36.767535    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-49ccf\" (UniqueName: \"kubernetes.io/projected/a229b032-a66e-4eda-9b0a-587219ae1c2b-kube-api-access-49ccf\") pod \"a229b032-a66e-4eda-9b0a-587219ae1c2b\" (UID: \"a229b032-a66e-4eda-9b0a-587219ae1c2b\") "
Jul 26 16:23:36.767703 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:23:36.767614    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a229b032-a66e-4eda-9b0a-587219ae1c2b-maas-api-tls\") pod \"a229b032-a66e-4eda-9b0a-587219ae1c2b\" (UID: \"a229b032-a66e-4eda-9b0a-587219ae1c2b\") "
Jul 26 16:23:36.767876 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:23:36.767850    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a229b032-a66e-4eda-9b0a-587219ae1c2b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a229b032-a66e-4eda-9b0a-587219ae1c2b" (UID: "a229b032-a66e-4eda-9b0a-587219ae1c2b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 26 16:23:36.769622 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:23:36.769598    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a229b032-a66e-4eda-9b0a-587219ae1c2b-kube-api-access-49ccf" (OuterVolumeSpecName: "kube-api-access-49ccf") pod "a229b032-a66e-4eda-9b0a-587219ae1c2b" (UID: "a229b032-a66e-4eda-9b0a-587219ae1c2b"). InnerVolumeSpecName "kube-api-access-49ccf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 26 16:23:36.769622 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:23:36.769613    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a229b032-a66e-4eda-9b0a-587219ae1c2b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a229b032-a66e-4eda-9b0a-587219ae1c2b" (UID: "a229b032-a66e-4eda-9b0a-587219ae1c2b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 26 16:23:36.869015 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:23:36.868929    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4xdbj\" (UniqueName: \"kubernetes.io/projected/9514fe11-e115-416e-bebd-7cea1bc5af4a-kube-api-access-4xdbj\") pod \"9514fe11-e115-416e-bebd-7cea1bc5af4a\" (UID: \"9514fe11-e115-416e-bebd-7cea1bc5af4a\") "
Jul 26 16:23:36.869015 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:23:36.868991    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53bf2ad2-8173-49f4-9056-0f575eb57695-plugins-config-volume\") pod \"53bf2ad2-8173-49f4-9056-0f575eb57695\" (UID: \"53bf2ad2-8173-49f4-9056-0f575eb57695\") "
Jul 26 16:23:36.869209 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:23:36.869047    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9514fe11-e115-416e-bebd-7cea1bc5af4a-plugins-config-volume\") pod \"9514fe11-e115-416e-bebd-7cea1bc5af4a\" (UID: \"9514fe11-e115-416e-bebd-7cea1bc5af4a\") "
Jul 26 16:23:36.869209 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:23:36.869063    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k6829\" (UniqueName: \"kubernetes.io/projected/53bf2ad2-8173-49f4-9056-0f575eb57695-kube-api-access-k6829\") pod \"53bf2ad2-8173-49f4-9056-0f575eb57695\" (UID: \"53bf2ad2-8173-49f4-9056-0f575eb57695\") "
Jul 26 16:23:36.869281 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:23:36.869266    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49ccf\" (UniqueName: \"kubernetes.io/projected/a229b032-a66e-4eda-9b0a-587219ae1c2b-kube-api-access-49ccf\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:23:36.869281 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:23:36.869277    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a229b032-a66e-4eda-9b0a-587219ae1c2b-maas-api-tls\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:23:36.869343 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:23:36.869286    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a229b032-a66e-4eda-9b0a-587219ae1c2b-openshift-service-ca\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:23:36.869425 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:23:36.869401    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53bf2ad2-8173-49f4-9056-0f575eb57695-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "53bf2ad2-8173-49f4-9056-0f575eb57695" (UID: "53bf2ad2-8173-49f4-9056-0f575eb57695"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 26 16:23:36.869485 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:23:36.869463    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9514fe11-e115-416e-bebd-7cea1bc5af4a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9514fe11-e115-416e-bebd-7cea1bc5af4a" (UID: "9514fe11-e115-416e-bebd-7cea1bc5af4a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 26 16:23:36.871165 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:23:36.871143    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9514fe11-e115-416e-bebd-7cea1bc5af4a-kube-api-access-4xdbj" (OuterVolumeSpecName: "kube-api-access-4xdbj") pod "9514fe11-e115-416e-bebd-7cea1bc5af4a" (UID: "9514fe11-e115-416e-bebd-7cea1bc5af4a"). InnerVolumeSpecName "kube-api-access-4xdbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 26 16:23:36.871250 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:23:36.871196    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53bf2ad2-8173-49f4-9056-0f575eb57695-kube-api-access-k6829" (OuterVolumeSpecName: "kube-api-access-k6829") pod "53bf2ad2-8173-49f4-9056-0f575eb57695" (UID: "53bf2ad2-8173-49f4-9056-0f575eb57695"). InnerVolumeSpecName "kube-api-access-k6829". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 26 16:23:36.970809 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:23:36.970750    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xdbj\" (UniqueName: \"kubernetes.io/projected/9514fe11-e115-416e-bebd-7cea1bc5af4a-kube-api-access-4xdbj\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:23:36.970809 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:23:36.970805    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53bf2ad2-8173-49f4-9056-0f575eb57695-plugins-config-volume\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:23:36.970809 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:23:36.970815    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9514fe11-e115-416e-bebd-7cea1bc5af4a-plugins-config-volume\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:23:36.971056 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:23:36.970824    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6829\" (UniqueName: \"kubernetes.io/projected/53bf2ad2-8173-49f4-9056-0f575eb57695-kube-api-access-k6829\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:23:37.146311 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:23:37.146218    2563 generic.go:356] "Generic (PLEG): container finished" podID="9514fe11-e115-416e-bebd-7cea1bc5af4a" containerID="14f17233cdc3a64367b8739f87cf51ca1b555c2dbce76fd1296f45829fa83397" exitCode=0
Jul 26 16:23:37.146311 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:23:37.146264    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-631b3d03-757985dbbb-mv4l6"
Jul 26 16:23:37.146554 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:23:37.146304    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-631b3d03-757985dbbb-mv4l6" event={"ID":"9514fe11-e115-416e-bebd-7cea1bc5af4a","Type":"ContainerDied","Data":"14f17233cdc3a64367b8739f87cf51ca1b555c2dbce76fd1296f45829fa83397"}
Jul 26 16:23:37.146554 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:23:37.146355    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-631b3d03-757985dbbb-mv4l6" event={"ID":"9514fe11-e115-416e-bebd-7cea1bc5af4a","Type":"ContainerDied","Data":"2deb882cf7f3b332bcfca780f8fa55fdd8adbfcfd73e170cb4e7222384c6e49c"}
Jul 26 16:23:37.146554 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:23:37.146383    2563 scope.go:117] "RemoveContainer" containerID="14f17233cdc3a64367b8739f87cf51ca1b555c2dbce76fd1296f45829fa83397"
Jul 26 16:23:37.147609 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:23:37.147590    2563 generic.go:356] "Generic (PLEG): container finished" podID="53bf2ad2-8173-49f4-9056-0f575eb57695" containerID="1b2f1a41c29ffc851a090c089c40d5080e798e89bbac305f13d3a0d7b2876b55" exitCode=0
Jul 26 16:23:37.147701 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:23:37.147632    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-631b3d03-6454c8fdd4-dsvlp"
Jul 26 16:23:37.147701 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:23:37.147682    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-631b3d03-6454c8fdd4-dsvlp" event={"ID":"53bf2ad2-8173-49f4-9056-0f575eb57695","Type":"ContainerDied","Data":"1b2f1a41c29ffc851a090c089c40d5080e798e89bbac305f13d3a0d7b2876b55"}
Jul 26 16:23:37.147843 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:23:37.147738    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-631b3d03-6454c8fdd4-dsvlp" event={"ID":"53bf2ad2-8173-49f4-9056-0f575eb57695","Type":"ContainerDied","Data":"425288cde71ebc981da6f92a9ce8fc2f8f56251bb0079a2d39e7160fbc5ba7c8"}
Jul 26 16:23:37.149183 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:23:37.149158    2563 generic.go:356] "Generic (PLEG): container finished" podID="a229b032-a66e-4eda-9b0a-587219ae1c2b" containerID="4a29898cd6283bc89cff841facdc106053a42695001dcd0cf91741b0522c4ebd" exitCode=0
Jul 26 16:23:37.149290 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:23:37.149214    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-631b3d03-67f5d946d6-f5t7n"
Jul 26 16:23:37.149353 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:23:37.149213    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-631b3d03-67f5d946d6-f5t7n" event={"ID":"a229b032-a66e-4eda-9b0a-587219ae1c2b","Type":"ContainerDied","Data":"4a29898cd6283bc89cff841facdc106053a42695001dcd0cf91741b0522c4ebd"}
Jul 26 16:23:37.149353 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:23:37.149336    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-631b3d03-67f5d946d6-f5t7n" event={"ID":"a229b032-a66e-4eda-9b0a-587219ae1c2b","Type":"ContainerDied","Data":"45e23d4c23ea65221b2d015f65e47a6c0ce6a046765d956a7c6b1205ab79b1fb"}
Jul 26 16:23:37.156291 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:23:37.156260    2563 scope.go:117] "RemoveContainer" containerID="14f17233cdc3a64367b8739f87cf51ca1b555c2dbce76fd1296f45829fa83397"
Jul 26 16:23:37.156542 ip-10-0-133-204 kubenswrapper[2563]: E0726 16:23:37.156522    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14f17233cdc3a64367b8739f87cf51ca1b555c2dbce76fd1296f45829fa83397\": container with ID starting with 14f17233cdc3a64367b8739f87cf51ca1b555c2dbce76fd1296f45829fa83397 not found: ID does not exist" containerID="14f17233cdc3a64367b8739f87cf51ca1b555c2dbce76fd1296f45829fa83397"
Jul 26 16:23:37.156637 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:23:37.156548    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14f17233cdc3a64367b8739f87cf51ca1b555c2dbce76fd1296f45829fa83397"} err="failed to get container status \"14f17233cdc3a64367b8739f87cf51ca1b555c2dbce76fd1296f45829fa83397\": rpc error: code = NotFound desc = could not find container \"14f17233cdc3a64367b8739f87cf51ca1b555c2dbce76fd1296f45829fa83397\": container with ID starting with 14f17233cdc3a64367b8739f87cf51ca1b555c2dbce76fd1296f45829fa83397 not found: ID does not exist"
Jul 26 16:23:37.156637 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:23:37.156564    2563 scope.go:117] "RemoveContainer" containerID="1b2f1a41c29ffc851a090c089c40d5080e798e89bbac305f13d3a0d7b2876b55"
Jul 26 16:23:37.165192 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:23:37.165175    2563 scope.go:117] "RemoveContainer" containerID="1b2f1a41c29ffc851a090c089c40d5080e798e89bbac305f13d3a0d7b2876b55"
Jul 26 16:23:37.165460 ip-10-0-133-204 kubenswrapper[2563]: E0726 16:23:37.165441    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b2f1a41c29ffc851a090c089c40d5080e798e89bbac305f13d3a0d7b2876b55\": container with ID starting with 1b2f1a41c29ffc851a090c089c40d5080e798e89bbac305f13d3a0d7b2876b55 not found: ID does not exist" containerID="1b2f1a41c29ffc851a090c089c40d5080e798e89bbac305f13d3a0d7b2876b55"
Jul 26 16:23:37.165524 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:23:37.165468    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b2f1a41c29ffc851a090c089c40d5080e798e89bbac305f13d3a0d7b2876b55"} err="failed to get container status \"1b2f1a41c29ffc851a090c089c40d5080e798e89bbac305f13d3a0d7b2876b55\": rpc error: code = NotFound desc = could not find container \"1b2f1a41c29ffc851a090c089c40d5080e798e89bbac305f13d3a0d7b2876b55\": container with ID starting with 1b2f1a41c29ffc851a090c089c40d5080e798e89bbac305f13d3a0d7b2876b55 not found: ID does not exist"
Jul 26 16:23:37.165524 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:23:37.165485    2563 scope.go:117] "RemoveContainer" containerID="4a29898cd6283bc89cff841facdc106053a42695001dcd0cf91741b0522c4ebd"
Jul 26 16:23:37.173226 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:23:37.173211    2563 scope.go:117] "RemoveContainer" containerID="4a29898cd6283bc89cff841facdc106053a42695001dcd0cf91741b0522c4ebd"
Jul 26 16:23:37.173472 ip-10-0-133-204 kubenswrapper[2563]: E0726 16:23:37.173452    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a29898cd6283bc89cff841facdc106053a42695001dcd0cf91741b0522c4ebd\": container with ID starting with 4a29898cd6283bc89cff841facdc106053a42695001dcd0cf91741b0522c4ebd not found: ID does not exist" containerID="4a29898cd6283bc89cff841facdc106053a42695001dcd0cf91741b0522c4ebd"
Jul 26 16:23:37.173512 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:23:37.173484    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a29898cd6283bc89cff841facdc106053a42695001dcd0cf91741b0522c4ebd"} err="failed to get container status \"4a29898cd6283bc89cff841facdc106053a42695001dcd0cf91741b0522c4ebd\": rpc error: code = NotFound desc = could not find container \"4a29898cd6283bc89cff841facdc106053a42695001dcd0cf91741b0522c4ebd\": container with ID starting with 4a29898cd6283bc89cff841facdc106053a42695001dcd0cf91741b0522c4ebd not found: ID does not exist"
Jul 26 16:23:37.178550 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:23:37.178531    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-631b3d03-6454c8fdd4-dsvlp"]
Jul 26 16:23:37.185625 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:23:37.185606    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-631b3d03-6454c8fdd4-dsvlp"]
Jul 26 16:23:37.205707 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:23:37.205686    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-631b3d03-757985dbbb-mv4l6"]
Jul 26 16:23:37.210227 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:23:37.210197    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-631b3d03-757985dbbb-mv4l6"]
Jul 26 16:23:37.228712 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:23:37.228686    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-631b3d03-67f5d946d6-f5t7n"]
Jul 26 16:23:37.244696 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:23:37.244673    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-631b3d03-67f5d946d6-f5t7n"]
Jul 26 16:23:38.380594 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:23:38.380563    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53bf2ad2-8173-49f4-9056-0f575eb57695" path="/var/lib/kubelet/pods/53bf2ad2-8173-49f4-9056-0f575eb57695/volumes"
Jul 26 16:23:38.380994 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:23:38.380939    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9514fe11-e115-416e-bebd-7cea1bc5af4a" path="/var/lib/kubelet/pods/9514fe11-e115-416e-bebd-7cea1bc5af4a/volumes"
Jul 26 16:23:38.381278 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:23:38.381265    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a229b032-a66e-4eda-9b0a-587219ae1c2b" path="/var/lib/kubelet/pods/a229b032-a66e-4eda-9b0a-587219ae1c2b/volumes"
Jul 26 16:23:40.889595 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:23:40.889560    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:23:44.325805 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:23:44.325772    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:23:49.350678 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:23:49.350641    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:24:16.347596 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:24:16.347563    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:24:25.733101 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:24:25.733067    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:24:26.379310 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:24:26.379232    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:24:31.945910 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:24:31.945816    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:24:46.497562 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:24:46.497528    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:24:46.597904 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:24:46.597872    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:24:46.804885 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:24:46.804807    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:24:47.025557 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:24:47.025525    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:24:48.223386 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:24:48.223350    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:24:48.326710 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:24:48.326676    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:24:48.420694 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:24:48.420657    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:24:49.182291 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:24:49.182262    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:24:49.281562 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:24:49.281530    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:24:49.382146 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:24:49.382112    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:24:50.803735 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:24:50.803700    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:24:56.674426 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:24:56.674390    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:24:56.771873 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:24:56.771841    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:24:57.347553 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:24:57.347515    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:25:03.747334 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:03.747299    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:25:03.859878 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:03.859814    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:25:03.952512 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:03.952476    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:25:14.481098 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:14.481041    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:25:14.842111 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:14.842025    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:25:35.851132 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:35.851096    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:25:41.967043 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:41.966989    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:25:42.068515 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:42.068454    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:25:42.332874 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:42.332842    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:25:42.671720 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:42.671611    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:25:42.806749 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:42.806694    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:25:45.895205 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:45.895149    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:25:46.000435 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:46.000400    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:25:46.143094 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:46.143057    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-54fb7c-6c4656b5bc-464p5"]
Jul 26 16:25:46.143730 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:46.143713    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53bf2ad2-8173-49f4-9056-0f575eb57695" containerName="payload-pre-processing"
Jul 26 16:25:46.143811 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:46.143734    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="53bf2ad2-8173-49f4-9056-0f575eb57695" containerName="payload-pre-processing"
Jul 26 16:25:46.143811 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:46.143755    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9514fe11-e115-416e-bebd-7cea1bc5af4a" containerName="payload-processing"
Jul 26 16:25:46.143811 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:46.143786    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9514fe11-e115-416e-bebd-7cea1bc5af4a" containerName="payload-processing"
Jul 26 16:25:46.143811 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:46.143801    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a229b032-a66e-4eda-9b0a-587219ae1c2b" containerName="maas-api"
Jul 26 16:25:46.143811 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:46.143809    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a229b032-a66e-4eda-9b0a-587219ae1c2b" containerName="maas-api"
Jul 26 16:25:46.143985 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:46.143895    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a229b032-a66e-4eda-9b0a-587219ae1c2b" containerName="maas-api"
Jul 26 16:25:46.143985 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:46.143905    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9514fe11-e115-416e-bebd-7cea1bc5af4a" containerName="payload-processing"
Jul 26 16:25:46.143985 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:46.143916    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="53bf2ad2-8173-49f4-9056-0f575eb57695" containerName="payload-pre-processing"
Jul 26 16:25:46.147568 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:46.147512    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-54fb7c-6c4656b5bc-464p5"
Jul 26 16:25:46.150804 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:46.150785    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-54fb7c-serving-cert\""
Jul 26 16:25:46.158106 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:46.158071    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-54fb7c-6c4656b5bc-464p5"]
Jul 26 16:25:46.166722 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:46.166694    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49be7d1a-3b51-45f2-a727-0b04c9bb25eb-maas-api-tls\") pod \"maas-api-e2e-inf-a-54fb7c-6c4656b5bc-464p5\" (UID: \"49be7d1a-3b51-45f2-a727-0b04c9bb25eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-54fb7c-6c4656b5bc-464p5"
Jul 26 16:25:46.166892 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:46.166784    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49be7d1a-3b51-45f2-a727-0b04c9bb25eb-openshift-service-ca\") pod \"maas-api-e2e-inf-a-54fb7c-6c4656b5bc-464p5\" (UID: \"49be7d1a-3b51-45f2-a727-0b04c9bb25eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-54fb7c-6c4656b5bc-464p5"
Jul 26 16:25:46.166892 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:46.166814    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvtqx\" (UniqueName: \"kubernetes.io/projected/49be7d1a-3b51-45f2-a727-0b04c9bb25eb-kube-api-access-rvtqx\") pod \"maas-api-e2e-inf-a-54fb7c-6c4656b5bc-464p5\" (UID: \"49be7d1a-3b51-45f2-a727-0b04c9bb25eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-54fb7c-6c4656b5bc-464p5"
Jul 26 16:25:46.187708 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:46.187630    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:25:46.268266 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:46.268234    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49be7d1a-3b51-45f2-a727-0b04c9bb25eb-openshift-service-ca\") pod \"maas-api-e2e-inf-a-54fb7c-6c4656b5bc-464p5\" (UID: \"49be7d1a-3b51-45f2-a727-0b04c9bb25eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-54fb7c-6c4656b5bc-464p5"
Jul 26 16:25:46.268266 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:46.268269    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvtqx\" (UniqueName: \"kubernetes.io/projected/49be7d1a-3b51-45f2-a727-0b04c9bb25eb-kube-api-access-rvtqx\") pod \"maas-api-e2e-inf-a-54fb7c-6c4656b5bc-464p5\" (UID: \"49be7d1a-3b51-45f2-a727-0b04c9bb25eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-54fb7c-6c4656b5bc-464p5"
Jul 26 16:25:46.268493 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:46.268331    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49be7d1a-3b51-45f2-a727-0b04c9bb25eb-maas-api-tls\") pod \"maas-api-e2e-inf-a-54fb7c-6c4656b5bc-464p5\" (UID: \"49be7d1a-3b51-45f2-a727-0b04c9bb25eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-54fb7c-6c4656b5bc-464p5"
Jul 26 16:25:46.268493 ip-10-0-133-204 kubenswrapper[2563]: E0726 16:25:46.268453    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-54fb7c-serving-cert: secret "maas-api-e2e-inf-a-54fb7c-serving-cert" not found
Jul 26 16:25:46.268578 ip-10-0-133-204 kubenswrapper[2563]: E0726 16:25:46.268556    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49be7d1a-3b51-45f2-a727-0b04c9bb25eb-maas-api-tls podName:49be7d1a-3b51-45f2-a727-0b04c9bb25eb nodeName:}" failed. No retries permitted until 2026-07-26 16:25:46.768533363 +0000 UTC m=+2138.872759167 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/49be7d1a-3b51-45f2-a727-0b04c9bb25eb-maas-api-tls") pod "maas-api-e2e-inf-a-54fb7c-6c4656b5bc-464p5" (UID: "49be7d1a-3b51-45f2-a727-0b04c9bb25eb") : secret "maas-api-e2e-inf-a-54fb7c-serving-cert" not found
Jul 26 16:25:46.268981 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:46.268961    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49be7d1a-3b51-45f2-a727-0b04c9bb25eb-openshift-service-ca\") pod \"maas-api-e2e-inf-a-54fb7c-6c4656b5bc-464p5\" (UID: \"49be7d1a-3b51-45f2-a727-0b04c9bb25eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-54fb7c-6c4656b5bc-464p5"
Jul 26 16:25:46.278477 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:46.278450    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvtqx\" (UniqueName: \"kubernetes.io/projected/49be7d1a-3b51-45f2-a727-0b04c9bb25eb-kube-api-access-rvtqx\") pod \"maas-api-e2e-inf-a-54fb7c-6c4656b5bc-464p5\" (UID: \"49be7d1a-3b51-45f2-a727-0b04c9bb25eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-54fb7c-6c4656b5bc-464p5"
Jul 26 16:25:46.419374 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:46.419289    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:25:46.773969 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:46.773940    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49be7d1a-3b51-45f2-a727-0b04c9bb25eb-maas-api-tls\") pod \"maas-api-e2e-inf-a-54fb7c-6c4656b5bc-464p5\" (UID: \"49be7d1a-3b51-45f2-a727-0b04c9bb25eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-54fb7c-6c4656b5bc-464p5"
Jul 26 16:25:46.776383 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:46.776361    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49be7d1a-3b51-45f2-a727-0b04c9bb25eb-maas-api-tls\") pod \"maas-api-e2e-inf-a-54fb7c-6c4656b5bc-464p5\" (UID: \"49be7d1a-3b51-45f2-a727-0b04c9bb25eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-54fb7c-6c4656b5bc-464p5"
Jul 26 16:25:46.788498 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:46.788458    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:25:46.882631 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:46.882596    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-54fb7c-6b847468d4-x59qn"]
Jul 26 16:25:46.886695 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:46.886667    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-54fb7c-6b847468d4-x59qn"
Jul 26 16:25:46.888065 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:46.888038    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:25:46.889248 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:46.889224    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-54fb7c-dockercfg-psxkd\""
Jul 26 16:25:46.889361 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:46.889314    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-54fb7c\""
Jul 26 16:25:46.900019 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:46.899998    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-54fb7c-6b847468d4-x59qn"]
Jul 26 16:25:46.919685 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:46.919649    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-54fb7c-8676fbf568-vzpdc"]
Jul 26 16:25:46.923293 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:46.923269    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-54fb7c-8676fbf568-vzpdc"
Jul 26 16:25:46.936698 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:46.936670    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-54fb7c-8676fbf568-vzpdc"]
Jul 26 16:25:46.975807 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:46.975776    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgqhs\" (UniqueName: \"kubernetes.io/projected/d900cc4f-2706-4858-87e2-2b82eefe25e7-kube-api-access-cgqhs\") pod \"payload-pre-processing-e2e-inf-a-54fb7c-6b847468d4-x59qn\" (UID: \"d900cc4f-2706-4858-87e2-2b82eefe25e7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-54fb7c-6b847468d4-x59qn"
Jul 26 16:25:46.975970 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:46.975819    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f865fca9-9a1d-46ec-85b9-a8e023f06c8f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-54fb7c-8676fbf568-vzpdc\" (UID: \"f865fca9-9a1d-46ec-85b9-a8e023f06c8f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-54fb7c-8676fbf568-vzpdc"
Jul 26 16:25:46.975970 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:46.975886    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjxn7\" (UniqueName: \"kubernetes.io/projected/f865fca9-9a1d-46ec-85b9-a8e023f06c8f-kube-api-access-pjxn7\") pod \"payload-processing-e2e-inf-a-54fb7c-8676fbf568-vzpdc\" (UID: \"f865fca9-9a1d-46ec-85b9-a8e023f06c8f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-54fb7c-8676fbf568-vzpdc"
Jul 26 16:25:46.975970 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:46.975943    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d900cc4f-2706-4858-87e2-2b82eefe25e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-54fb7c-6b847468d4-x59qn\" (UID: \"d900cc4f-2706-4858-87e2-2b82eefe25e7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-54fb7c-6b847468d4-x59qn"
Jul 26 16:25:47.061419 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:47.061337    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-54fb7c-6c4656b5bc-464p5"
Jul 26 16:25:47.076897 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:47.076871    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjxn7\" (UniqueName: \"kubernetes.io/projected/f865fca9-9a1d-46ec-85b9-a8e023f06c8f-kube-api-access-pjxn7\") pod \"payload-processing-e2e-inf-a-54fb7c-8676fbf568-vzpdc\" (UID: \"f865fca9-9a1d-46ec-85b9-a8e023f06c8f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-54fb7c-8676fbf568-vzpdc"
Jul 26 16:25:47.077062 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:47.076929    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d900cc4f-2706-4858-87e2-2b82eefe25e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-54fb7c-6b847468d4-x59qn\" (UID: \"d900cc4f-2706-4858-87e2-2b82eefe25e7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-54fb7c-6b847468d4-x59qn"
Jul 26 16:25:47.077062 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:47.076982    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgqhs\" (UniqueName: \"kubernetes.io/projected/d900cc4f-2706-4858-87e2-2b82eefe25e7-kube-api-access-cgqhs\") pod \"payload-pre-processing-e2e-inf-a-54fb7c-6b847468d4-x59qn\" (UID: \"d900cc4f-2706-4858-87e2-2b82eefe25e7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-54fb7c-6b847468d4-x59qn"
Jul 26 16:25:47.077062 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:47.077022    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f865fca9-9a1d-46ec-85b9-a8e023f06c8f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-54fb7c-8676fbf568-vzpdc\" (UID: \"f865fca9-9a1d-46ec-85b9-a8e023f06c8f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-54fb7c-8676fbf568-vzpdc"
Jul 26 16:25:47.077877 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:47.077855    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f865fca9-9a1d-46ec-85b9-a8e023f06c8f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-54fb7c-8676fbf568-vzpdc\" (UID: \"f865fca9-9a1d-46ec-85b9-a8e023f06c8f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-54fb7c-8676fbf568-vzpdc"
Jul 26 16:25:47.078003 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:47.077855    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d900cc4f-2706-4858-87e2-2b82eefe25e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-54fb7c-6b847468d4-x59qn\" (UID: \"d900cc4f-2706-4858-87e2-2b82eefe25e7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-54fb7c-6b847468d4-x59qn"
Jul 26 16:25:47.086672 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:47.086618    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgqhs\" (UniqueName: \"kubernetes.io/projected/d900cc4f-2706-4858-87e2-2b82eefe25e7-kube-api-access-cgqhs\") pod \"payload-pre-processing-e2e-inf-a-54fb7c-6b847468d4-x59qn\" (UID: \"d900cc4f-2706-4858-87e2-2b82eefe25e7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-54fb7c-6b847468d4-x59qn"
Jul 26 16:25:47.088826 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:47.088745    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjxn7\" (UniqueName: \"kubernetes.io/projected/f865fca9-9a1d-46ec-85b9-a8e023f06c8f-kube-api-access-pjxn7\") pod \"payload-processing-e2e-inf-a-54fb7c-8676fbf568-vzpdc\" (UID: \"f865fca9-9a1d-46ec-85b9-a8e023f06c8f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-54fb7c-8676fbf568-vzpdc"
Jul 26 16:25:47.199598 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:47.199567    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-54fb7c-6b847468d4-x59qn"
Jul 26 16:25:47.221403 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:47.221378    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-54fb7c-6c4656b5bc-464p5"]
Jul 26 16:25:47.223886 ip-10-0-133-204 kubenswrapper[2563]: W0726 16:25:47.223855    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49be7d1a_3b51_45f2_a727_0b04c9bb25eb.slice/crio-f2e57dc60571535df3742fb372bc4012ccb60624430c48556041912df210d20e WatchSource:0}: Error finding container f2e57dc60571535df3742fb372bc4012ccb60624430c48556041912df210d20e: Status 404 returned error can't find the container with id f2e57dc60571535df3742fb372bc4012ccb60624430c48556041912df210d20e
Jul 26 16:25:47.233994 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:47.233970    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-54fb7c-8676fbf568-vzpdc"
Jul 26 16:25:47.367586 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:47.367535    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-54fb7c-6b847468d4-x59qn"]
Jul 26 16:25:47.369789 ip-10-0-133-204 kubenswrapper[2563]: W0726 16:25:47.369743    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd900cc4f_2706_4858_87e2_2b82eefe25e7.slice/crio-fb5c4705313e024e05366f96fa826698080629cec88a77b4a606e3e55a9f025b WatchSource:0}: Error finding container fb5c4705313e024e05366f96fa826698080629cec88a77b4a606e3e55a9f025b: Status 404 returned error can't find the container with id fb5c4705313e024e05366f96fa826698080629cec88a77b4a606e3e55a9f025b
Jul 26 16:25:47.392516 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:47.392493    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-54fb7c-8676fbf568-vzpdc"]
Jul 26 16:25:47.395322 ip-10-0-133-204 kubenswrapper[2563]: W0726 16:25:47.395291    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf865fca9_9a1d_46ec_85b9_a8e023f06c8f.slice/crio-77a49adaaca80ddc20b8ec9905bcb5c8a99aa37ffab5dd5e060bb093610ee2ae WatchSource:0}: Error finding container 77a49adaaca80ddc20b8ec9905bcb5c8a99aa37ffab5dd5e060bb093610ee2ae: Status 404 returned error can't find the container with id 77a49adaaca80ddc20b8ec9905bcb5c8a99aa37ffab5dd5e060bb093610ee2ae
Jul 26 16:25:47.769242 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:47.769209    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-54fb7c-8676fbf568-vzpdc" event={"ID":"f865fca9-9a1d-46ec-85b9-a8e023f06c8f","Type":"ContainerStarted","Data":"254d012548c8af503f7f6342e95c15289b52320f61d4e2f0f6980a39d8a2a8bf"}
Jul 26 16:25:47.769449 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:47.769265    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-54fb7c-8676fbf568-vzpdc" event={"ID":"f865fca9-9a1d-46ec-85b9-a8e023f06c8f","Type":"ContainerStarted","Data":"77a49adaaca80ddc20b8ec9905bcb5c8a99aa37ffab5dd5e060bb093610ee2ae"}
Jul 26 16:25:47.769449 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:47.769387    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-54fb7c-8676fbf568-vzpdc"
Jul 26 16:25:47.770788 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:47.770744    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-54fb7c-6b847468d4-x59qn" event={"ID":"d900cc4f-2706-4858-87e2-2b82eefe25e7","Type":"ContainerStarted","Data":"c663975183aa6864767041470651b9cf1eac0df5e553bac426ef416dea84c0d4"}
Jul 26 16:25:47.770915 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:47.770802    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-54fb7c-6b847468d4-x59qn" event={"ID":"d900cc4f-2706-4858-87e2-2b82eefe25e7","Type":"ContainerStarted","Data":"fb5c4705313e024e05366f96fa826698080629cec88a77b4a606e3e55a9f025b"}
Jul 26 16:25:47.770915 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:47.770828    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-54fb7c-6b847468d4-x59qn"
Jul 26 16:25:47.772156 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:47.772139    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-54fb7c-6c4656b5bc-464p5" event={"ID":"49be7d1a-3b51-45f2-a727-0b04c9bb25eb","Type":"ContainerStarted","Data":"8fb12dc92f9b02665fc973a17b240b6ef4e4ae822aa5991fc6fa75e846d99f2c"}
Jul 26 16:25:47.772353 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:47.772175    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-54fb7c-6c4656b5bc-464p5" event={"ID":"49be7d1a-3b51-45f2-a727-0b04c9bb25eb","Type":"ContainerStarted","Data":"f2e57dc60571535df3742fb372bc4012ccb60624430c48556041912df210d20e"}
Jul 26 16:25:47.772353 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:47.772213    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-54fb7c-6c4656b5bc-464p5"
Jul 26 16:25:47.789808 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:47.789732    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-54fb7c-8676fbf568-vzpdc" podStartSLOduration=1.789712231 podStartE2EDuration="1.789712231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:25:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:25:47.784963803 +0000 UTC m=+2139.889189613" watchObservedRunningTime="2026-07-26 16:25:47.789712231 +0000 UTC m=+2139.893938040"
Jul 26 16:25:47.803147 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:47.803094    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-54fb7c-6c4656b5bc-464p5" podStartSLOduration=1.409664628 podStartE2EDuration="1.803078122s" totalImagesPullingTime="393.413494ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:25:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:25:47.801382955 +0000 UTC m=+2139.905608778" watchObservedRunningTime="2026-07-26 16:25:47.803078122 +0000 UTC m=+2139.907303931"
Jul 26 16:25:47.819621 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:47.819571    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-54fb7c-6b847468d4-x59qn" podStartSLOduration=1.819554046 podStartE2EDuration="1.819554046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:25:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:25:47.817333764 +0000 UTC m=+2139.921559600" watchObservedRunningTime="2026-07-26 16:25:47.819554046 +0000 UTC m=+2139.923779854"
Jul 26 16:25:48.784353 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:48.784321    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:25:50.422281 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:50.422245    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:25:50.768085 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:50.768046    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:25:51.099178 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:51.099075    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:25:51.257687 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:51.257656    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:25:53.790751 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:53.790723    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-54fb7c-6c4656b5bc-464p5"
Jul 26 16:25:53.901511 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:53.901479    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:25:53.982010 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:53.981969    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3b85e1-7fd4f578b7-7xjsc"]
Jul 26 16:25:53.983665 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:53.983643    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:25:53.986533 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:53.986506    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3b85e1-7fd4f578b7-7xjsc"
Jul 26 16:25:53.989000 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:53.988981    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-3b85e1-serving-cert\""
Jul 26 16:25:53.994965 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:53.994945    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3b85e1-7fd4f578b7-7xjsc"]
Jul 26 16:25:54.046307 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:54.046209    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djhk8\" (UniqueName: \"kubernetes.io/projected/226246a4-2163-4e54-bcf0-06e54a205c4b-kube-api-access-djhk8\") pod \"maas-api-e2e-inf-b-3b85e1-7fd4f578b7-7xjsc\" (UID: \"226246a4-2163-4e54-bcf0-06e54a205c4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3b85e1-7fd4f578b7-7xjsc"
Jul 26 16:25:54.046482 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:54.046375    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/226246a4-2163-4e54-bcf0-06e54a205c4b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-3b85e1-7fd4f578b7-7xjsc\" (UID: \"226246a4-2163-4e54-bcf0-06e54a205c4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3b85e1-7fd4f578b7-7xjsc"
Jul 26 16:25:54.046482 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:54.046417    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/226246a4-2163-4e54-bcf0-06e54a205c4b-maas-api-tls\") pod \"maas-api-e2e-inf-b-3b85e1-7fd4f578b7-7xjsc\" (UID: \"226246a4-2163-4e54-bcf0-06e54a205c4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3b85e1-7fd4f578b7-7xjsc"
Jul 26 16:25:54.053959 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:54.053934    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:25:54.147282 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:54.147237    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/226246a4-2163-4e54-bcf0-06e54a205c4b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-3b85e1-7fd4f578b7-7xjsc\" (UID: \"226246a4-2163-4e54-bcf0-06e54a205c4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3b85e1-7fd4f578b7-7xjsc"
Jul 26 16:25:54.147282 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:54.147281    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/226246a4-2163-4e54-bcf0-06e54a205c4b-maas-api-tls\") pod \"maas-api-e2e-inf-b-3b85e1-7fd4f578b7-7xjsc\" (UID: \"226246a4-2163-4e54-bcf0-06e54a205c4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3b85e1-7fd4f578b7-7xjsc"
Jul 26 16:25:54.147556 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:54.147324    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djhk8\" (UniqueName: \"kubernetes.io/projected/226246a4-2163-4e54-bcf0-06e54a205c4b-kube-api-access-djhk8\") pod \"maas-api-e2e-inf-b-3b85e1-7fd4f578b7-7xjsc\" (UID: \"226246a4-2163-4e54-bcf0-06e54a205c4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3b85e1-7fd4f578b7-7xjsc"
Jul 26 16:25:54.147556 ip-10-0-133-204 kubenswrapper[2563]: E0726 16:25:54.147461    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-3b85e1-serving-cert: secret "maas-api-e2e-inf-b-3b85e1-serving-cert" not found
Jul 26 16:25:54.147556 ip-10-0-133-204 kubenswrapper[2563]: E0726 16:25:54.147540    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/226246a4-2163-4e54-bcf0-06e54a205c4b-maas-api-tls podName:226246a4-2163-4e54-bcf0-06e54a205c4b nodeName:}" failed. No retries permitted until 2026-07-26 16:25:54.647517696 +0000 UTC m=+2146.751743483 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/226246a4-2163-4e54-bcf0-06e54a205c4b-maas-api-tls") pod "maas-api-e2e-inf-b-3b85e1-7fd4f578b7-7xjsc" (UID: "226246a4-2163-4e54-bcf0-06e54a205c4b") : secret "maas-api-e2e-inf-b-3b85e1-serving-cert" not found
Jul 26 16:25:54.148051 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:54.148032    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/226246a4-2163-4e54-bcf0-06e54a205c4b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-3b85e1-7fd4f578b7-7xjsc\" (UID: \"226246a4-2163-4e54-bcf0-06e54a205c4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3b85e1-7fd4f578b7-7xjsc"
Jul 26 16:25:54.155990 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:54.155969    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djhk8\" (UniqueName: \"kubernetes.io/projected/226246a4-2163-4e54-bcf0-06e54a205c4b-kube-api-access-djhk8\") pod \"maas-api-e2e-inf-b-3b85e1-7fd4f578b7-7xjsc\" (UID: \"226246a4-2163-4e54-bcf0-06e54a205c4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3b85e1-7fd4f578b7-7xjsc"
Jul 26 16:25:54.295587 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:54.295555    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:25:54.391635 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:54.391551    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:25:54.590244 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:54.590208    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:25:54.651255 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:54.651159    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/226246a4-2163-4e54-bcf0-06e54a205c4b-maas-api-tls\") pod \"maas-api-e2e-inf-b-3b85e1-7fd4f578b7-7xjsc\" (UID: \"226246a4-2163-4e54-bcf0-06e54a205c4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3b85e1-7fd4f578b7-7xjsc"
Jul 26 16:25:54.653717 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:54.653692    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/226246a4-2163-4e54-bcf0-06e54a205c4b-maas-api-tls\") pod \"maas-api-e2e-inf-b-3b85e1-7fd4f578b7-7xjsc\" (UID: \"226246a4-2163-4e54-bcf0-06e54a205c4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3b85e1-7fd4f578b7-7xjsc"
Jul 26 16:25:54.692172 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:54.692129    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:25:54.698929 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:54.698894    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3b85e1-6f59885748-687bl"]
Jul 26 16:25:54.704310 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:54.704278    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3b85e1-6f59885748-687bl"
Jul 26 16:25:54.706537 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:54.706510    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-3b85e1\""
Jul 26 16:25:54.709750 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:54.709715    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3b85e1-6f59885748-687bl"]
Jul 26 16:25:54.719163 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:54.719136    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3b85e1-6dc9fcf7d4-rjds6"]
Jul 26 16:25:54.722955 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:54.722923    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-3b85e1-6dc9fcf7d4-rjds6"
Jul 26 16:25:54.725252 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:54.725220    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-3b85e1-dockercfg-fqkl7\""
Jul 26 16:25:54.733092 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:54.732882    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3b85e1-6dc9fcf7d4-rjds6"]
Jul 26 16:25:54.752545 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:54.752509    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s94t\" (UniqueName: \"kubernetes.io/projected/18f5c217-f197-4d75-9538-a3c8727233a5-kube-api-access-8s94t\") pod \"payload-processing-e2e-inf-b-3b85e1-6dc9fcf7d4-rjds6\" (UID: \"18f5c217-f197-4d75-9538-a3c8727233a5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3b85e1-6dc9fcf7d4-rjds6"
Jul 26 16:25:54.752545 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:54.752544    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2bq6\" (UniqueName: \"kubernetes.io/projected/a70fb590-7499-439d-b551-e7af3d4ac4a3-kube-api-access-g2bq6\") pod \"payload-pre-processing-e2e-inf-b-3b85e1-6f59885748-687bl\" (UID: \"a70fb590-7499-439d-b551-e7af3d4ac4a3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3b85e1-6f59885748-687bl"
Jul 26 16:25:54.752860 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:54.752578    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18f5c217-f197-4d75-9538-a3c8727233a5-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-3b85e1-6dc9fcf7d4-rjds6\" (UID: \"18f5c217-f197-4d75-9538-a3c8727233a5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3b85e1-6dc9fcf7d4-rjds6"
Jul 26 16:25:54.752860 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:54.752682    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a70fb590-7499-439d-b551-e7af3d4ac4a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-3b85e1-6f59885748-687bl\" (UID: \"a70fb590-7499-439d-b551-e7af3d4ac4a3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3b85e1-6f59885748-687bl"
Jul 26 16:25:54.853496 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:54.853456    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a70fb590-7499-439d-b551-e7af3d4ac4a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-3b85e1-6f59885748-687bl\" (UID: \"a70fb590-7499-439d-b551-e7af3d4ac4a3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3b85e1-6f59885748-687bl"
Jul 26 16:25:54.854019 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:54.853557    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s94t\" (UniqueName: \"kubernetes.io/projected/18f5c217-f197-4d75-9538-a3c8727233a5-kube-api-access-8s94t\") pod \"payload-processing-e2e-inf-b-3b85e1-6dc9fcf7d4-rjds6\" (UID: \"18f5c217-f197-4d75-9538-a3c8727233a5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3b85e1-6dc9fcf7d4-rjds6"
Jul 26 16:25:54.854019 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:54.853579    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2bq6\" (UniqueName: \"kubernetes.io/projected/a70fb590-7499-439d-b551-e7af3d4ac4a3-kube-api-access-g2bq6\") pod \"payload-pre-processing-e2e-inf-b-3b85e1-6f59885748-687bl\" (UID: \"a70fb590-7499-439d-b551-e7af3d4ac4a3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3b85e1-6f59885748-687bl"
Jul 26 16:25:54.854019 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:54.853611    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18f5c217-f197-4d75-9538-a3c8727233a5-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-3b85e1-6dc9fcf7d4-rjds6\" (UID: \"18f5c217-f197-4d75-9538-a3c8727233a5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3b85e1-6dc9fcf7d4-rjds6"
Jul 26 16:25:54.854306 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:54.854284    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18f5c217-f197-4d75-9538-a3c8727233a5-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-3b85e1-6dc9fcf7d4-rjds6\" (UID: \"18f5c217-f197-4d75-9538-a3c8727233a5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3b85e1-6dc9fcf7d4-rjds6"
Jul 26 16:25:54.854384 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:54.854367    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a70fb590-7499-439d-b551-e7af3d4ac4a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-3b85e1-6f59885748-687bl\" (UID: \"a70fb590-7499-439d-b551-e7af3d4ac4a3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3b85e1-6f59885748-687bl"
Jul 26 16:25:54.862116 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:54.862079    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2bq6\" (UniqueName: \"kubernetes.io/projected/a70fb590-7499-439d-b551-e7af3d4ac4a3-kube-api-access-g2bq6\") pod \"payload-pre-processing-e2e-inf-b-3b85e1-6f59885748-687bl\" (UID: \"a70fb590-7499-439d-b551-e7af3d4ac4a3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3b85e1-6f59885748-687bl"
Jul 26 16:25:54.862258 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:54.862080    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s94t\" (UniqueName: \"kubernetes.io/projected/18f5c217-f197-4d75-9538-a3c8727233a5-kube-api-access-8s94t\") pod \"payload-processing-e2e-inf-b-3b85e1-6dc9fcf7d4-rjds6\" (UID: \"18f5c217-f197-4d75-9538-a3c8727233a5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3b85e1-6dc9fcf7d4-rjds6"
Jul 26 16:25:54.901838 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:54.901734    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3b85e1-7fd4f578b7-7xjsc"
Jul 26 16:25:55.018537 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:55.018501    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3b85e1-6f59885748-687bl"
Jul 26 16:25:55.037732 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:55.037698    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-3b85e1-6dc9fcf7d4-rjds6"
Jul 26 16:25:55.051624 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:55.051593    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3b85e1-7fd4f578b7-7xjsc"]
Jul 26 16:25:55.053290 ip-10-0-133-204 kubenswrapper[2563]: W0726 16:25:55.053265    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod226246a4_2163_4e54_bcf0_06e54a205c4b.slice/crio-e0a1ee85a93967187c3a0dc5437a76878dc5415c5921a21b79144c61a6208010 WatchSource:0}: Error finding container e0a1ee85a93967187c3a0dc5437a76878dc5415c5921a21b79144c61a6208010: Status 404 returned error can't find the container with id e0a1ee85a93967187c3a0dc5437a76878dc5415c5921a21b79144c61a6208010
Jul 26 16:25:55.181015 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:55.180816    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3b85e1-6f59885748-687bl"]
Jul 26 16:25:55.185298 ip-10-0-133-204 kubenswrapper[2563]: W0726 16:25:55.185154    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda70fb590_7499_439d_b551_e7af3d4ac4a3.slice/crio-d24160a59c521f19018cd1f87aa81a0e731600967ea2fe97fc0ebc65ce5e3ceb WatchSource:0}: Error finding container d24160a59c521f19018cd1f87aa81a0e731600967ea2fe97fc0ebc65ce5e3ceb: Status 404 returned error can't find the container with id d24160a59c521f19018cd1f87aa81a0e731600967ea2fe97fc0ebc65ce5e3ceb
Jul 26 16:25:55.212740 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:55.212712    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3b85e1-6dc9fcf7d4-rjds6"]
Jul 26 16:25:55.214199 ip-10-0-133-204 kubenswrapper[2563]: W0726 16:25:55.214171    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18f5c217_f197_4d75_9538_a3c8727233a5.slice/crio-8d7ddc288ea1c56b164912882f9513183b6b1669832eba0a96be8204d85ca35b WatchSource:0}: Error finding container 8d7ddc288ea1c56b164912882f9513183b6b1669832eba0a96be8204d85ca35b: Status 404 returned error can't find the container with id 8d7ddc288ea1c56b164912882f9513183b6b1669832eba0a96be8204d85ca35b
Jul 26 16:25:55.825019 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:55.824983    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-3b85e1-6dc9fcf7d4-rjds6" event={"ID":"18f5c217-f197-4d75-9538-a3c8727233a5","Type":"ContainerStarted","Data":"aa6f020a61a14d8600e901b3cc59356deb79dd32a21ab4d525cf7519985bac93"}
Jul 26 16:25:55.825231 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:55.825047    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-3b85e1-6dc9fcf7d4-rjds6" event={"ID":"18f5c217-f197-4d75-9538-a3c8727233a5","Type":"ContainerStarted","Data":"8d7ddc288ea1c56b164912882f9513183b6b1669832eba0a96be8204d85ca35b"}
Jul 26 16:25:55.825231 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:55.825170    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-3b85e1-6dc9fcf7d4-rjds6"
Jul 26 16:25:55.826552 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:55.826527    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3b85e1-6f59885748-687bl" event={"ID":"a70fb590-7499-439d-b551-e7af3d4ac4a3","Type":"ContainerStarted","Data":"8eb1f80ae81fa3527912198991d08d9a910399138ebe1d56ad7d307f8a1464e7"}
Jul 26 16:25:55.826700 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:55.826578    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3b85e1-6f59885748-687bl"
Jul 26 16:25:55.826700 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:55.826600    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3b85e1-6f59885748-687bl" event={"ID":"a70fb590-7499-439d-b551-e7af3d4ac4a3","Type":"ContainerStarted","Data":"d24160a59c521f19018cd1f87aa81a0e731600967ea2fe97fc0ebc65ce5e3ceb"}
Jul 26 16:25:55.828613 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:55.828586    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3b85e1-7fd4f578b7-7xjsc" event={"ID":"226246a4-2163-4e54-bcf0-06e54a205c4b","Type":"ContainerStarted","Data":"63bdbf9c703ffddb11b201f22ac39dfa9466e48eaf37a193b2222dac38d972fe"}
Jul 26 16:25:55.828737 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:55.828651    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3b85e1-7fd4f578b7-7xjsc"
Jul 26 16:25:55.828737 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:55.828675    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3b85e1-7fd4f578b7-7xjsc" event={"ID":"226246a4-2163-4e54-bcf0-06e54a205c4b","Type":"ContainerStarted","Data":"e0a1ee85a93967187c3a0dc5437a76878dc5415c5921a21b79144c61a6208010"}
Jul 26 16:25:55.856506 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:55.856439    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3b85e1-6f59885748-687bl" podStartSLOduration=1.856421041 podStartE2EDuration="1.856421041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:25:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:25:55.855726155 +0000 UTC m=+2147.959951966" watchObservedRunningTime="2026-07-26 16:25:55.856421041 +0000 UTC m=+2147.960646902"
Jul 26 16:25:55.857036 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:55.856548    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-3b85e1-6dc9fcf7d4-rjds6" podStartSLOduration=1.8565412079999999 podStartE2EDuration="1.856541208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:25:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:25:55.840264818 +0000 UTC m=+2147.944490628" watchObservedRunningTime="2026-07-26 16:25:55.856541208 +0000 UTC m=+2147.960767019"
Jul 26 16:25:55.879302 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:55.879240    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3b85e1-7fd4f578b7-7xjsc" podStartSLOduration=2.547059447 podStartE2EDuration="2.879217171s" totalImagesPullingTime="332.157724ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:25:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:25:55.875746494 +0000 UTC m=+2147.979972320" watchObservedRunningTime="2026-07-26 16:25:55.879217171 +0000 UTC m=+2147.983442981"
Jul 26 16:25:58.433239 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:58.433204    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:25:58.786569 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:58.786524    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-54fb7c-8676fbf568-vzpdc"
Jul 26 16:25:58.786786 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:25:58.786610    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-54fb7c-6b847468d4-x59qn"
Jul 26 16:26:01.839302 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:26:01.839218    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3b85e1-7fd4f578b7-7xjsc"
Jul 26 16:26:06.835917 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:26:06.835883    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3b85e1-6f59885748-687bl"
Jul 26 16:26:06.836477 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:26:06.836046    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-3b85e1-6dc9fcf7d4-rjds6"
Jul 26 16:26:09.378732 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:26:09.378701    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:26:38.275584 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:26:38.275544    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:27:13.370106 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:13.370071    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:27:18.203562 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:18.203453    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:27:19.197169 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:19.197129    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:27:29.739570 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:29.739529    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-54fb7c-6c4656b5bc-464p5"]
Jul 26 16:27:29.740215 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:29.739801    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-54fb7c-6c4656b5bc-464p5" podUID="49be7d1a-3b51-45f2-a727-0b04c9bb25eb" containerName="maas-api" containerID="cri-o://8fb12dc92f9b02665fc973a17b240b6ef4e4ae822aa5991fc6fa75e846d99f2c" gracePeriod=30
Jul 26 16:27:29.765741 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:29.765716    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:27:29.796029 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:29.795996    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-54fb7c-8676fbf568-vzpdc"]
Jul 26 16:27:29.796239 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:29.796214    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-54fb7c-8676fbf568-vzpdc" podUID="f865fca9-9a1d-46ec-85b9-a8e023f06c8f" containerName="payload-processing" containerID="cri-o://254d012548c8af503f7f6342e95c15289b52320f61d4e2f0f6980a39d8a2a8bf" gracePeriod=30
Jul 26 16:27:29.804320 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:29.804292    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-54fb7c-6b847468d4-x59qn"]
Jul 26 16:27:29.804510 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:29.804488    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-54fb7c-6b847468d4-x59qn" podUID="d900cc4f-2706-4858-87e2-2b82eefe25e7" containerName="payload-pre-processing" containerID="cri-o://c663975183aa6864767041470651b9cf1eac0df5e553bac426ef416dea84c0d4" gracePeriod=30
Jul 26 16:27:30.067674 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:30.067640    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-54fb7c-6c4656b5bc-464p5"
Jul 26 16:27:30.113867 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:30.113840    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-54fb7c-8676fbf568-vzpdc"
Jul 26 16:27:30.124951 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:30.124928    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-54fb7c-6b847468d4-x59qn"
Jul 26 16:27:30.160890 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:30.160862    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49be7d1a-3b51-45f2-a727-0b04c9bb25eb-openshift-service-ca\") pod \"49be7d1a-3b51-45f2-a727-0b04c9bb25eb\" (UID: \"49be7d1a-3b51-45f2-a727-0b04c9bb25eb\") "
Jul 26 16:27:30.161082 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:30.160897    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49be7d1a-3b51-45f2-a727-0b04c9bb25eb-maas-api-tls\") pod \"49be7d1a-3b51-45f2-a727-0b04c9bb25eb\" (UID: \"49be7d1a-3b51-45f2-a727-0b04c9bb25eb\") "
Jul 26 16:27:30.161082 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:30.160928    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rvtqx\" (UniqueName: \"kubernetes.io/projected/49be7d1a-3b51-45f2-a727-0b04c9bb25eb-kube-api-access-rvtqx\") pod \"49be7d1a-3b51-45f2-a727-0b04c9bb25eb\" (UID: \"49be7d1a-3b51-45f2-a727-0b04c9bb25eb\") "
Jul 26 16:27:30.161250 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:30.161227    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49be7d1a-3b51-45f2-a727-0b04c9bb25eb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "49be7d1a-3b51-45f2-a727-0b04c9bb25eb" (UID: "49be7d1a-3b51-45f2-a727-0b04c9bb25eb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 26 16:27:30.162976 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:30.162951    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49be7d1a-3b51-45f2-a727-0b04c9bb25eb-kube-api-access-rvtqx" (OuterVolumeSpecName: "kube-api-access-rvtqx") pod "49be7d1a-3b51-45f2-a727-0b04c9bb25eb" (UID: "49be7d1a-3b51-45f2-a727-0b04c9bb25eb"). InnerVolumeSpecName "kube-api-access-rvtqx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 26 16:27:30.163094 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:30.162997    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49be7d1a-3b51-45f2-a727-0b04c9bb25eb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "49be7d1a-3b51-45f2-a727-0b04c9bb25eb" (UID: "49be7d1a-3b51-45f2-a727-0b04c9bb25eb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 26 16:27:30.261991 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:30.261895    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cgqhs\" (UniqueName: \"kubernetes.io/projected/d900cc4f-2706-4858-87e2-2b82eefe25e7-kube-api-access-cgqhs\") pod \"d900cc4f-2706-4858-87e2-2b82eefe25e7\" (UID: \"d900cc4f-2706-4858-87e2-2b82eefe25e7\") "
Jul 26 16:27:30.261991 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:30.261934    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f865fca9-9a1d-46ec-85b9-a8e023f06c8f-plugins-config-volume\") pod \"f865fca9-9a1d-46ec-85b9-a8e023f06c8f\" (UID: \"f865fca9-9a1d-46ec-85b9-a8e023f06c8f\") "
Jul 26 16:27:30.261991 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:30.261962    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d900cc4f-2706-4858-87e2-2b82eefe25e7-plugins-config-volume\") pod \"d900cc4f-2706-4858-87e2-2b82eefe25e7\" (UID: \"d900cc4f-2706-4858-87e2-2b82eefe25e7\") "
Jul 26 16:27:30.262292 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:30.262042    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pjxn7\" (UniqueName: \"kubernetes.io/projected/f865fca9-9a1d-46ec-85b9-a8e023f06c8f-kube-api-access-pjxn7\") pod \"f865fca9-9a1d-46ec-85b9-a8e023f06c8f\" (UID: \"f865fca9-9a1d-46ec-85b9-a8e023f06c8f\") "
Jul 26 16:27:30.262292 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:30.262223    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvtqx\" (UniqueName: \"kubernetes.io/projected/49be7d1a-3b51-45f2-a727-0b04c9bb25eb-kube-api-access-rvtqx\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:27:30.262292 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:30.262234    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49be7d1a-3b51-45f2-a727-0b04c9bb25eb-openshift-service-ca\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:27:30.262292 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:30.262245    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49be7d1a-3b51-45f2-a727-0b04c9bb25eb-maas-api-tls\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:27:30.262467 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:30.262336    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f865fca9-9a1d-46ec-85b9-a8e023f06c8f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f865fca9-9a1d-46ec-85b9-a8e023f06c8f" (UID: "f865fca9-9a1d-46ec-85b9-a8e023f06c8f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 26 16:27:30.262467 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:30.262340    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d900cc4f-2706-4858-87e2-2b82eefe25e7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d900cc4f-2706-4858-87e2-2b82eefe25e7" (UID: "d900cc4f-2706-4858-87e2-2b82eefe25e7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 26 16:27:30.263992 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:30.263966    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d900cc4f-2706-4858-87e2-2b82eefe25e7-kube-api-access-cgqhs" (OuterVolumeSpecName: "kube-api-access-cgqhs") pod "d900cc4f-2706-4858-87e2-2b82eefe25e7" (UID: "d900cc4f-2706-4858-87e2-2b82eefe25e7"). InnerVolumeSpecName "kube-api-access-cgqhs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 26 16:27:30.264098 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:30.264004    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f865fca9-9a1d-46ec-85b9-a8e023f06c8f-kube-api-access-pjxn7" (OuterVolumeSpecName: "kube-api-access-pjxn7") pod "f865fca9-9a1d-46ec-85b9-a8e023f06c8f" (UID: "f865fca9-9a1d-46ec-85b9-a8e023f06c8f"). InnerVolumeSpecName "kube-api-access-pjxn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 26 16:27:30.304544 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:30.304512    2563 generic.go:356] "Generic (PLEG): container finished" podID="49be7d1a-3b51-45f2-a727-0b04c9bb25eb" containerID="8fb12dc92f9b02665fc973a17b240b6ef4e4ae822aa5991fc6fa75e846d99f2c" exitCode=0
Jul 26 16:27:30.304705 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:30.304558    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-54fb7c-6c4656b5bc-464p5"
Jul 26 16:27:30.304705 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:30.304591    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-54fb7c-6c4656b5bc-464p5" event={"ID":"49be7d1a-3b51-45f2-a727-0b04c9bb25eb","Type":"ContainerDied","Data":"8fb12dc92f9b02665fc973a17b240b6ef4e4ae822aa5991fc6fa75e846d99f2c"}
Jul 26 16:27:30.304705 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:30.304639    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-54fb7c-6c4656b5bc-464p5" event={"ID":"49be7d1a-3b51-45f2-a727-0b04c9bb25eb","Type":"ContainerDied","Data":"f2e57dc60571535df3742fb372bc4012ccb60624430c48556041912df210d20e"}
Jul 26 16:27:30.304705 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:30.304670    2563 scope.go:117] "RemoveContainer" containerID="8fb12dc92f9b02665fc973a17b240b6ef4e4ae822aa5991fc6fa75e846d99f2c"
Jul 26 16:27:30.305896 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:30.305877    2563 generic.go:356] "Generic (PLEG): container finished" podID="f865fca9-9a1d-46ec-85b9-a8e023f06c8f" containerID="254d012548c8af503f7f6342e95c15289b52320f61d4e2f0f6980a39d8a2a8bf" exitCode=0
Jul 26 16:27:30.305978 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:30.305920    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-54fb7c-8676fbf568-vzpdc"
Jul 26 16:27:30.306039 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:30.305967    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-54fb7c-8676fbf568-vzpdc" event={"ID":"f865fca9-9a1d-46ec-85b9-a8e023f06c8f","Type":"ContainerDied","Data":"254d012548c8af503f7f6342e95c15289b52320f61d4e2f0f6980a39d8a2a8bf"}
Jul 26 16:27:30.306039 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:30.306018    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-54fb7c-8676fbf568-vzpdc" event={"ID":"f865fca9-9a1d-46ec-85b9-a8e023f06c8f","Type":"ContainerDied","Data":"77a49adaaca80ddc20b8ec9905bcb5c8a99aa37ffab5dd5e060bb093610ee2ae"}
Jul 26 16:27:30.307512 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:30.307488    2563 generic.go:356] "Generic (PLEG): container finished" podID="d900cc4f-2706-4858-87e2-2b82eefe25e7" containerID="c663975183aa6864767041470651b9cf1eac0df5e553bac426ef416dea84c0d4" exitCode=0
Jul 26 16:27:30.307614 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:30.307562    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-54fb7c-6b847468d4-x59qn" event={"ID":"d900cc4f-2706-4858-87e2-2b82eefe25e7","Type":"ContainerDied","Data":"c663975183aa6864767041470651b9cf1eac0df5e553bac426ef416dea84c0d4"}
Jul 26 16:27:30.307614 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:30.307579    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-54fb7c-6b847468d4-x59qn"
Jul 26 16:27:30.307614 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:30.307596    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-54fb7c-6b847468d4-x59qn" event={"ID":"d900cc4f-2706-4858-87e2-2b82eefe25e7","Type":"ContainerDied","Data":"fb5c4705313e024e05366f96fa826698080629cec88a77b4a606e3e55a9f025b"}
Jul 26 16:27:30.314067 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:30.313937    2563 scope.go:117] "RemoveContainer" containerID="8fb12dc92f9b02665fc973a17b240b6ef4e4ae822aa5991fc6fa75e846d99f2c"
Jul 26 16:27:30.314310 ip-10-0-133-204 kubenswrapper[2563]: E0726 16:27:30.314280    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fb12dc92f9b02665fc973a17b240b6ef4e4ae822aa5991fc6fa75e846d99f2c\": container with ID starting with 8fb12dc92f9b02665fc973a17b240b6ef4e4ae822aa5991fc6fa75e846d99f2c not found: ID does not exist" containerID="8fb12dc92f9b02665fc973a17b240b6ef4e4ae822aa5991fc6fa75e846d99f2c"
Jul 26 16:27:30.314411 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:30.314315    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fb12dc92f9b02665fc973a17b240b6ef4e4ae822aa5991fc6fa75e846d99f2c"} err="failed to get container status \"8fb12dc92f9b02665fc973a17b240b6ef4e4ae822aa5991fc6fa75e846d99f2c\": rpc error: code = NotFound desc = could not find container \"8fb12dc92f9b02665fc973a17b240b6ef4e4ae822aa5991fc6fa75e846d99f2c\": container with ID starting with 8fb12dc92f9b02665fc973a17b240b6ef4e4ae822aa5991fc6fa75e846d99f2c not found: ID does not exist"
Jul 26 16:27:30.314411 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:30.314334    2563 scope.go:117] "RemoveContainer" containerID="254d012548c8af503f7f6342e95c15289b52320f61d4e2f0f6980a39d8a2a8bf"
Jul 26 16:27:30.323794 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:30.323751    2563 scope.go:117] "RemoveContainer" containerID="254d012548c8af503f7f6342e95c15289b52320f61d4e2f0f6980a39d8a2a8bf"
Jul 26 16:27:30.324058 ip-10-0-133-204 kubenswrapper[2563]: E0726 16:27:30.324040    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"254d012548c8af503f7f6342e95c15289b52320f61d4e2f0f6980a39d8a2a8bf\": container with ID starting with 254d012548c8af503f7f6342e95c15289b52320f61d4e2f0f6980a39d8a2a8bf not found: ID does not exist" containerID="254d012548c8af503f7f6342e95c15289b52320f61d4e2f0f6980a39d8a2a8bf"
Jul 26 16:27:30.324107 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:30.324065    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"254d012548c8af503f7f6342e95c15289b52320f61d4e2f0f6980a39d8a2a8bf"} err="failed to get container status \"254d012548c8af503f7f6342e95c15289b52320f61d4e2f0f6980a39d8a2a8bf\": rpc error: code = NotFound desc = could not find container \"254d012548c8af503f7f6342e95c15289b52320f61d4e2f0f6980a39d8a2a8bf\": container with ID starting with 254d012548c8af503f7f6342e95c15289b52320f61d4e2f0f6980a39d8a2a8bf not found: ID does not exist"
Jul 26 16:27:30.324107 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:30.324081    2563 scope.go:117] "RemoveContainer" containerID="c663975183aa6864767041470651b9cf1eac0df5e553bac426ef416dea84c0d4"
Jul 26 16:27:30.332160 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:30.332140    2563 scope.go:117] "RemoveContainer" containerID="c663975183aa6864767041470651b9cf1eac0df5e553bac426ef416dea84c0d4"
Jul 26 16:27:30.332421 ip-10-0-133-204 kubenswrapper[2563]: E0726 16:27:30.332403    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c663975183aa6864767041470651b9cf1eac0df5e553bac426ef416dea84c0d4\": container with ID starting with c663975183aa6864767041470651b9cf1eac0df5e553bac426ef416dea84c0d4 not found: ID does not exist" containerID="c663975183aa6864767041470651b9cf1eac0df5e553bac426ef416dea84c0d4"
Jul 26 16:27:30.332487 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:30.332429    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c663975183aa6864767041470651b9cf1eac0df5e553bac426ef416dea84c0d4"} err="failed to get container status \"c663975183aa6864767041470651b9cf1eac0df5e553bac426ef416dea84c0d4\": rpc error: code = NotFound desc = could not find container \"c663975183aa6864767041470651b9cf1eac0df5e553bac426ef416dea84c0d4\": container with ID starting with c663975183aa6864767041470651b9cf1eac0df5e553bac426ef416dea84c0d4 not found: ID does not exist"
Jul 26 16:27:30.332713 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:30.332694    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-54fb7c-6c4656b5bc-464p5"]
Jul 26 16:27:30.336230 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:30.336210    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-54fb7c-6c4656b5bc-464p5"]
Jul 26 16:27:30.346437 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:30.346416    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-54fb7c-8676fbf568-vzpdc"]
Jul 26 16:27:30.349928 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:30.349908    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-54fb7c-8676fbf568-vzpdc"]
Jul 26 16:27:30.360491 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:30.360462    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-54fb7c-6b847468d4-x59qn"]
Jul 26 16:27:30.363497 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:30.363472    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgqhs\" (UniqueName: \"kubernetes.io/projected/d900cc4f-2706-4858-87e2-2b82eefe25e7-kube-api-access-cgqhs\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:27:30.363497 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:30.363494    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f865fca9-9a1d-46ec-85b9-a8e023f06c8f-plugins-config-volume\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:27:30.363717 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:30.363504    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d900cc4f-2706-4858-87e2-2b82eefe25e7-plugins-config-volume\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:27:30.363717 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:30.363513    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjxn7\" (UniqueName: \"kubernetes.io/projected/f865fca9-9a1d-46ec-85b9-a8e023f06c8f-kube-api-access-pjxn7\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:27:30.363972 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:30.363955    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-54fb7c-6b847468d4-x59qn"]
Jul 26 16:27:30.373380 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:30.373348    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49be7d1a-3b51-45f2-a727-0b04c9bb25eb" path="/var/lib/kubelet/pods/49be7d1a-3b51-45f2-a727-0b04c9bb25eb/volumes"
Jul 26 16:27:30.373805 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:30.373788    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d900cc4f-2706-4858-87e2-2b82eefe25e7" path="/var/lib/kubelet/pods/d900cc4f-2706-4858-87e2-2b82eefe25e7/volumes"
Jul 26 16:27:30.374171 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:30.374155    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f865fca9-9a1d-46ec-85b9-a8e023f06c8f" path="/var/lib/kubelet/pods/f865fca9-9a1d-46ec-85b9-a8e023f06c8f/volumes"
Jul 26 16:27:30.838855 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:30.838820    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:27:34.397184 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:34.397146    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:27:34.817144 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:34.817112    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:27:40.577657 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:40.577611    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:27:40.680986 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:40.680950    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:27:40.995607 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:40.995569    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:27:41.246561 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:41.246479    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:27:51.842191 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:51.842156    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3b85e1-7fd4f578b7-7xjsc"]
Jul 26 16:27:51.843009 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:51.842403    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3b85e1-7fd4f578b7-7xjsc" podUID="226246a4-2163-4e54-bcf0-06e54a205c4b" containerName="maas-api" containerID="cri-o://63bdbf9c703ffddb11b201f22ac39dfa9466e48eaf37a193b2222dac38d972fe" gracePeriod=30
Jul 26 16:27:51.875103 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:51.875067    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:27:51.923695 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:51.923557    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3b85e1-6f59885748-687bl"]
Jul 26 16:27:51.923965 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:51.923843    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3b85e1-6f59885748-687bl" podUID="a70fb590-7499-439d-b551-e7af3d4ac4a3" containerName="payload-pre-processing" containerID="cri-o://8eb1f80ae81fa3527912198991d08d9a910399138ebe1d56ad7d307f8a1464e7" gracePeriod=30
Jul 26 16:27:51.926538 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:51.926509    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3b85e1-6dc9fcf7d4-rjds6"]
Jul 26 16:27:51.926731 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:51.926709    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-3b85e1-6dc9fcf7d4-rjds6" podUID="18f5c217-f197-4d75-9538-a3c8727233a5" containerName="payload-processing" containerID="cri-o://aa6f020a61a14d8600e901b3cc59356deb79dd32a21ab4d525cf7519985bac93" gracePeriod=30
Jul 26 16:27:51.970131 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:51.970098    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:27:52.033643 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:52.033607    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:27:52.224075 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:52.223807    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-3b85e1-6dc9fcf7d4-rjds6"
Jul 26 16:27:52.228335 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:52.228313    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3b85e1-6f59885748-687bl"
Jul 26 16:27:52.273469 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:52.273438    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18f5c217-f197-4d75-9538-a3c8727233a5-plugins-config-volume\") pod \"18f5c217-f197-4d75-9538-a3c8727233a5\" (UID: \"18f5c217-f197-4d75-9538-a3c8727233a5\") "
Jul 26 16:27:52.273650 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:52.273540    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8s94t\" (UniqueName: \"kubernetes.io/projected/18f5c217-f197-4d75-9538-a3c8727233a5-kube-api-access-8s94t\") pod \"18f5c217-f197-4d75-9538-a3c8727233a5\" (UID: \"18f5c217-f197-4d75-9538-a3c8727233a5\") "
Jul 26 16:27:52.273650 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:52.273613    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a70fb590-7499-439d-b551-e7af3d4ac4a3-plugins-config-volume\") pod \"a70fb590-7499-439d-b551-e7af3d4ac4a3\" (UID: \"a70fb590-7499-439d-b551-e7af3d4ac4a3\") "
Jul 26 16:27:52.273812 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:52.273705    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g2bq6\" (UniqueName: \"kubernetes.io/projected/a70fb590-7499-439d-b551-e7af3d4ac4a3-kube-api-access-g2bq6\") pod \"a70fb590-7499-439d-b551-e7af3d4ac4a3\" (UID: \"a70fb590-7499-439d-b551-e7af3d4ac4a3\") "
Jul 26 16:27:52.273903 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:52.273878    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18f5c217-f197-4d75-9538-a3c8727233a5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "18f5c217-f197-4d75-9538-a3c8727233a5" (UID: "18f5c217-f197-4d75-9538-a3c8727233a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 26 16:27:52.274140 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:52.274111    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a70fb590-7499-439d-b551-e7af3d4ac4a3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a70fb590-7499-439d-b551-e7af3d4ac4a3" (UID: "a70fb590-7499-439d-b551-e7af3d4ac4a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 26 16:27:52.274265 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:52.274136    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18f5c217-f197-4d75-9538-a3c8727233a5-plugins-config-volume\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:27:52.275719 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:52.275697    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a70fb590-7499-439d-b551-e7af3d4ac4a3-kube-api-access-g2bq6" (OuterVolumeSpecName: "kube-api-access-g2bq6") pod "a70fb590-7499-439d-b551-e7af3d4ac4a3" (UID: "a70fb590-7499-439d-b551-e7af3d4ac4a3"). InnerVolumeSpecName "kube-api-access-g2bq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 26 16:27:52.276021 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:52.276004    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18f5c217-f197-4d75-9538-a3c8727233a5-kube-api-access-8s94t" (OuterVolumeSpecName: "kube-api-access-8s94t") pod "18f5c217-f197-4d75-9538-a3c8727233a5" (UID: "18f5c217-f197-4d75-9538-a3c8727233a5"). InnerVolumeSpecName "kube-api-access-8s94t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 26 16:27:52.292001 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:52.291976    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3b85e1-7fd4f578b7-7xjsc"
Jul 26 16:27:52.374611 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:52.374540    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/226246a4-2163-4e54-bcf0-06e54a205c4b-openshift-service-ca\") pod \"226246a4-2163-4e54-bcf0-06e54a205c4b\" (UID: \"226246a4-2163-4e54-bcf0-06e54a205c4b\") "
Jul 26 16:27:52.374611 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:52.374571    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/226246a4-2163-4e54-bcf0-06e54a205c4b-maas-api-tls\") pod \"226246a4-2163-4e54-bcf0-06e54a205c4b\" (UID: \"226246a4-2163-4e54-bcf0-06e54a205c4b\") "
Jul 26 16:27:52.374808 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:52.374630    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-djhk8\" (UniqueName: \"kubernetes.io/projected/226246a4-2163-4e54-bcf0-06e54a205c4b-kube-api-access-djhk8\") pod \"226246a4-2163-4e54-bcf0-06e54a205c4b\" (UID: \"226246a4-2163-4e54-bcf0-06e54a205c4b\") "
Jul 26 16:27:52.374884 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:52.374865    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8s94t\" (UniqueName: \"kubernetes.io/projected/18f5c217-f197-4d75-9538-a3c8727233a5-kube-api-access-8s94t\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:27:52.374956 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:52.374888    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a70fb590-7499-439d-b551-e7af3d4ac4a3-plugins-config-volume\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:27:52.374956 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:52.374901    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2bq6\" (UniqueName: \"kubernetes.io/projected/a70fb590-7499-439d-b551-e7af3d4ac4a3-kube-api-access-g2bq6\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:27:52.374956 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:52.374930    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/226246a4-2163-4e54-bcf0-06e54a205c4b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "226246a4-2163-4e54-bcf0-06e54a205c4b" (UID: "226246a4-2163-4e54-bcf0-06e54a205c4b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 26 16:27:52.376717 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:52.376684    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/226246a4-2163-4e54-bcf0-06e54a205c4b-kube-api-access-djhk8" (OuterVolumeSpecName: "kube-api-access-djhk8") pod "226246a4-2163-4e54-bcf0-06e54a205c4b" (UID: "226246a4-2163-4e54-bcf0-06e54a205c4b"). InnerVolumeSpecName "kube-api-access-djhk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 26 16:27:52.376717 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:52.376707    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/226246a4-2163-4e54-bcf0-06e54a205c4b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "226246a4-2163-4e54-bcf0-06e54a205c4b" (UID: "226246a4-2163-4e54-bcf0-06e54a205c4b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 26 16:27:52.419663 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:52.419632    2563 generic.go:356] "Generic (PLEG): container finished" podID="18f5c217-f197-4d75-9538-a3c8727233a5" containerID="aa6f020a61a14d8600e901b3cc59356deb79dd32a21ab4d525cf7519985bac93" exitCode=0
Jul 26 16:27:52.419833 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:52.419697    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-3b85e1-6dc9fcf7d4-rjds6"
Jul 26 16:27:52.419833 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:52.419711    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-3b85e1-6dc9fcf7d4-rjds6" event={"ID":"18f5c217-f197-4d75-9538-a3c8727233a5","Type":"ContainerDied","Data":"aa6f020a61a14d8600e901b3cc59356deb79dd32a21ab4d525cf7519985bac93"}
Jul 26 16:27:52.419833 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:52.419809    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-3b85e1-6dc9fcf7d4-rjds6" event={"ID":"18f5c217-f197-4d75-9538-a3c8727233a5","Type":"ContainerDied","Data":"8d7ddc288ea1c56b164912882f9513183b6b1669832eba0a96be8204d85ca35b"}
Jul 26 16:27:52.419992 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:52.419834    2563 scope.go:117] "RemoveContainer" containerID="aa6f020a61a14d8600e901b3cc59356deb79dd32a21ab4d525cf7519985bac93"
Jul 26 16:27:52.421017 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:52.420998    2563 generic.go:356] "Generic (PLEG): container finished" podID="a70fb590-7499-439d-b551-e7af3d4ac4a3" containerID="8eb1f80ae81fa3527912198991d08d9a910399138ebe1d56ad7d307f8a1464e7" exitCode=0
Jul 26 16:27:52.421113 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:52.421068    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3b85e1-6f59885748-687bl"
Jul 26 16:27:52.421113 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:52.421077    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3b85e1-6f59885748-687bl" event={"ID":"a70fb590-7499-439d-b551-e7af3d4ac4a3","Type":"ContainerDied","Data":"8eb1f80ae81fa3527912198991d08d9a910399138ebe1d56ad7d307f8a1464e7"}
Jul 26 16:27:52.421233 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:52.421117    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3b85e1-6f59885748-687bl" event={"ID":"a70fb590-7499-439d-b551-e7af3d4ac4a3","Type":"ContainerDied","Data":"d24160a59c521f19018cd1f87aa81a0e731600967ea2fe97fc0ebc65ce5e3ceb"}
Jul 26 16:27:52.422598 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:52.422577    2563 generic.go:356] "Generic (PLEG): container finished" podID="226246a4-2163-4e54-bcf0-06e54a205c4b" containerID="63bdbf9c703ffddb11b201f22ac39dfa9466e48eaf37a193b2222dac38d972fe" exitCode=0
Jul 26 16:27:52.422697 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:52.422646    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3b85e1-7fd4f578b7-7xjsc" event={"ID":"226246a4-2163-4e54-bcf0-06e54a205c4b","Type":"ContainerDied","Data":"63bdbf9c703ffddb11b201f22ac39dfa9466e48eaf37a193b2222dac38d972fe"}
Jul 26 16:27:52.422748 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:52.422690    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3b85e1-7fd4f578b7-7xjsc" event={"ID":"226246a4-2163-4e54-bcf0-06e54a205c4b","Type":"ContainerDied","Data":"e0a1ee85a93967187c3a0dc5437a76878dc5415c5921a21b79144c61a6208010"}
Jul 26 16:27:52.422748 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:52.422651    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3b85e1-7fd4f578b7-7xjsc"
Jul 26 16:27:52.430381 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:52.430364    2563 scope.go:117] "RemoveContainer" containerID="aa6f020a61a14d8600e901b3cc59356deb79dd32a21ab4d525cf7519985bac93"
Jul 26 16:27:52.430649 ip-10-0-133-204 kubenswrapper[2563]: E0726 16:27:52.430632    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa6f020a61a14d8600e901b3cc59356deb79dd32a21ab4d525cf7519985bac93\": container with ID starting with aa6f020a61a14d8600e901b3cc59356deb79dd32a21ab4d525cf7519985bac93 not found: ID does not exist" containerID="aa6f020a61a14d8600e901b3cc59356deb79dd32a21ab4d525cf7519985bac93"
Jul 26 16:27:52.430709 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:52.430658    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa6f020a61a14d8600e901b3cc59356deb79dd32a21ab4d525cf7519985bac93"} err="failed to get container status \"aa6f020a61a14d8600e901b3cc59356deb79dd32a21ab4d525cf7519985bac93\": rpc error: code = NotFound desc = could not find container \"aa6f020a61a14d8600e901b3cc59356deb79dd32a21ab4d525cf7519985bac93\": container with ID starting with aa6f020a61a14d8600e901b3cc59356deb79dd32a21ab4d525cf7519985bac93 not found: ID does not exist"
Jul 26 16:27:52.430709 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:52.430674    2563 scope.go:117] "RemoveContainer" containerID="8eb1f80ae81fa3527912198991d08d9a910399138ebe1d56ad7d307f8a1464e7"
Jul 26 16:27:52.438587 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:52.438570    2563 scope.go:117] "RemoveContainer" containerID="8eb1f80ae81fa3527912198991d08d9a910399138ebe1d56ad7d307f8a1464e7"
Jul 26 16:27:52.438846 ip-10-0-133-204 kubenswrapper[2563]: E0726 16:27:52.438827    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8eb1f80ae81fa3527912198991d08d9a910399138ebe1d56ad7d307f8a1464e7\": container with ID starting with 8eb1f80ae81fa3527912198991d08d9a910399138ebe1d56ad7d307f8a1464e7 not found: ID does not exist" containerID="8eb1f80ae81fa3527912198991d08d9a910399138ebe1d56ad7d307f8a1464e7"
Jul 26 16:27:52.438956 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:52.438849    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8eb1f80ae81fa3527912198991d08d9a910399138ebe1d56ad7d307f8a1464e7"} err="failed to get container status \"8eb1f80ae81fa3527912198991d08d9a910399138ebe1d56ad7d307f8a1464e7\": rpc error: code = NotFound desc = could not find container \"8eb1f80ae81fa3527912198991d08d9a910399138ebe1d56ad7d307f8a1464e7\": container with ID starting with 8eb1f80ae81fa3527912198991d08d9a910399138ebe1d56ad7d307f8a1464e7 not found: ID does not exist"
Jul 26 16:27:52.438956 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:52.438865    2563 scope.go:117] "RemoveContainer" containerID="63bdbf9c703ffddb11b201f22ac39dfa9466e48eaf37a193b2222dac38d972fe"
Jul 26 16:27:52.441696 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:52.441677    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3b85e1-6f59885748-687bl"]
Jul 26 16:27:52.445454 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:52.445430    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3b85e1-6f59885748-687bl"]
Jul 26 16:27:52.448197 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:52.448178    2563 scope.go:117] "RemoveContainer" containerID="63bdbf9c703ffddb11b201f22ac39dfa9466e48eaf37a193b2222dac38d972fe"
Jul 26 16:27:52.448462 ip-10-0-133-204 kubenswrapper[2563]: E0726 16:27:52.448442    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63bdbf9c703ffddb11b201f22ac39dfa9466e48eaf37a193b2222dac38d972fe\": container with ID starting with 63bdbf9c703ffddb11b201f22ac39dfa9466e48eaf37a193b2222dac38d972fe not found: ID does not exist" containerID="63bdbf9c703ffddb11b201f22ac39dfa9466e48eaf37a193b2222dac38d972fe"
Jul 26 16:27:52.448566 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:52.448472    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63bdbf9c703ffddb11b201f22ac39dfa9466e48eaf37a193b2222dac38d972fe"} err="failed to get container status \"63bdbf9c703ffddb11b201f22ac39dfa9466e48eaf37a193b2222dac38d972fe\": rpc error: code = NotFound desc = could not find container \"63bdbf9c703ffddb11b201f22ac39dfa9466e48eaf37a193b2222dac38d972fe\": container with ID starting with 63bdbf9c703ffddb11b201f22ac39dfa9466e48eaf37a193b2222dac38d972fe not found: ID does not exist"
Jul 26 16:27:52.455624 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:52.455605    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3b85e1-6dc9fcf7d4-rjds6"]
Jul 26 16:27:52.460140 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:52.460120    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3b85e1-6dc9fcf7d4-rjds6"]
Jul 26 16:27:52.469684 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:52.469667    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3b85e1-7fd4f578b7-7xjsc"]
Jul 26 16:27:52.473712 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:52.473696    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3b85e1-7fd4f578b7-7xjsc"]
Jul 26 16:27:52.475366 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:52.475343    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/226246a4-2163-4e54-bcf0-06e54a205c4b-openshift-service-ca\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:27:52.475458 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:52.475371    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/226246a4-2163-4e54-bcf0-06e54a205c4b-maas-api-tls\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:27:52.475458 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:52.475387    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djhk8\" (UniqueName: \"kubernetes.io/projected/226246a4-2163-4e54-bcf0-06e54a205c4b-kube-api-access-djhk8\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:27:54.372973 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:54.372941    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18f5c217-f197-4d75-9538-a3c8727233a5" path="/var/lib/kubelet/pods/18f5c217-f197-4d75-9538-a3c8727233a5/volumes"
Jul 26 16:27:54.373346 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:54.373335    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="226246a4-2163-4e54-bcf0-06e54a205c4b" path="/var/lib/kubelet/pods/226246a4-2163-4e54-bcf0-06e54a205c4b/volumes"
Jul 26 16:27:54.373653 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:54.373640    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a70fb590-7499-439d-b551-e7af3d4ac4a3" path="/var/lib/kubelet/pods/a70fb590-7499-439d-b551-e7af3d4ac4a3/volumes"
Jul 26 16:27:56.404834 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:56.404784    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:27:56.807136 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:27:56.807102    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:28:02.690101 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:02.690058    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:28:02.758418 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:02.758381    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:28:02.981530 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:02.981501    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:28:03.220128 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:03.220098    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:28:03.604261 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:03.604227    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:28:03.955723 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:03.955607    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:28:04.094476 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:04.094400    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:28:06.673270 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:06.673236    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:28:06.779985 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:06.779929    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:28:06.788730 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:06.788695    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-948a62-54b9f5d999-qtwbm"]
Jul 26 16:28:06.789299 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:06.789283    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49be7d1a-3b51-45f2-a727-0b04c9bb25eb" containerName="maas-api"
Jul 26 16:28:06.789352 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:06.789303    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="49be7d1a-3b51-45f2-a727-0b04c9bb25eb" containerName="maas-api"
Jul 26 16:28:06.789352 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:06.789318    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a70fb590-7499-439d-b551-e7af3d4ac4a3" containerName="payload-pre-processing"
Jul 26 16:28:06.789352 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:06.789328    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a70fb590-7499-439d-b551-e7af3d4ac4a3" containerName="payload-pre-processing"
Jul 26 16:28:06.789352 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:06.789341    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f865fca9-9a1d-46ec-85b9-a8e023f06c8f" containerName="payload-processing"
Jul 26 16:28:06.789352 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:06.789350    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f865fca9-9a1d-46ec-85b9-a8e023f06c8f" containerName="payload-processing"
Jul 26 16:28:06.789501 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:06.789365    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18f5c217-f197-4d75-9538-a3c8727233a5" containerName="payload-processing"
Jul 26 16:28:06.789501 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:06.789373    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="18f5c217-f197-4d75-9538-a3c8727233a5" containerName="payload-processing"
Jul 26 16:28:06.789501 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:06.789386    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d900cc4f-2706-4858-87e2-2b82eefe25e7" containerName="payload-pre-processing"
Jul 26 16:28:06.789501 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:06.789394    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d900cc4f-2706-4858-87e2-2b82eefe25e7" containerName="payload-pre-processing"
Jul 26 16:28:06.789501 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:06.789406    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="226246a4-2163-4e54-bcf0-06e54a205c4b" containerName="maas-api"
Jul 26 16:28:06.789501 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:06.789414    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="226246a4-2163-4e54-bcf0-06e54a205c4b" containerName="maas-api"
Jul 26 16:28:06.789673 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:06.789510    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f865fca9-9a1d-46ec-85b9-a8e023f06c8f" containerName="payload-processing"
Jul 26 16:28:06.789673 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:06.789521    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a70fb590-7499-439d-b551-e7af3d4ac4a3" containerName="payload-pre-processing"
Jul 26 16:28:06.789673 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:06.789531    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="18f5c217-f197-4d75-9538-a3c8727233a5" containerName="payload-processing"
Jul 26 16:28:06.789673 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:06.789540    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d900cc4f-2706-4858-87e2-2b82eefe25e7" containerName="payload-pre-processing"
Jul 26 16:28:06.789673 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:06.789549    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="226246a4-2163-4e54-bcf0-06e54a205c4b" containerName="maas-api"
Jul 26 16:28:06.789673 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:06.789557    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="49be7d1a-3b51-45f2-a727-0b04c9bb25eb" containerName="maas-api"
Jul 26 16:28:06.794510 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:06.794492    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-948a62-54b9f5d999-qtwbm"
Jul 26 16:28:06.796974 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:06.796953    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-948a62-serving-cert\""
Jul 26 16:28:06.806301 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:06.806272    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-948a62-54b9f5d999-qtwbm"]
Jul 26 16:28:06.872020 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:06.871985    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:28:06.906099 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:06.906065    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/085ec5e0-8366-47a6-abc1-e8f11109d7a0-openshift-service-ca\") pod \"maas-api-e2e-api-a-948a62-54b9f5d999-qtwbm\" (UID: \"085ec5e0-8366-47a6-abc1-e8f11109d7a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-948a62-54b9f5d999-qtwbm"
Jul 26 16:28:06.906300 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:06.906164    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/085ec5e0-8366-47a6-abc1-e8f11109d7a0-maas-api-tls\") pod \"maas-api-e2e-api-a-948a62-54b9f5d999-qtwbm\" (UID: \"085ec5e0-8366-47a6-abc1-e8f11109d7a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-948a62-54b9f5d999-qtwbm"
Jul 26 16:28:06.906300 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:06.906246    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jbz2\" (UniqueName: \"kubernetes.io/projected/085ec5e0-8366-47a6-abc1-e8f11109d7a0-kube-api-access-4jbz2\") pod \"maas-api-e2e-api-a-948a62-54b9f5d999-qtwbm\" (UID: \"085ec5e0-8366-47a6-abc1-e8f11109d7a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-948a62-54b9f5d999-qtwbm"
Jul 26 16:28:06.982036 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:06.982005    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:28:07.007254 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:07.007224    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/085ec5e0-8366-47a6-abc1-e8f11109d7a0-openshift-service-ca\") pod \"maas-api-e2e-api-a-948a62-54b9f5d999-qtwbm\" (UID: \"085ec5e0-8366-47a6-abc1-e8f11109d7a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-948a62-54b9f5d999-qtwbm"
Jul 26 16:28:07.007402 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:07.007260    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/085ec5e0-8366-47a6-abc1-e8f11109d7a0-maas-api-tls\") pod \"maas-api-e2e-api-a-948a62-54b9f5d999-qtwbm\" (UID: \"085ec5e0-8366-47a6-abc1-e8f11109d7a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-948a62-54b9f5d999-qtwbm"
Jul 26 16:28:07.007402 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:07.007308    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jbz2\" (UniqueName: \"kubernetes.io/projected/085ec5e0-8366-47a6-abc1-e8f11109d7a0-kube-api-access-4jbz2\") pod \"maas-api-e2e-api-a-948a62-54b9f5d999-qtwbm\" (UID: \"085ec5e0-8366-47a6-abc1-e8f11109d7a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-948a62-54b9f5d999-qtwbm"
Jul 26 16:28:07.008001 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:07.007981    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/085ec5e0-8366-47a6-abc1-e8f11109d7a0-openshift-service-ca\") pod \"maas-api-e2e-api-a-948a62-54b9f5d999-qtwbm\" (UID: \"085ec5e0-8366-47a6-abc1-e8f11109d7a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-948a62-54b9f5d999-qtwbm"
Jul 26 16:28:07.009734 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:07.009712    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/085ec5e0-8366-47a6-abc1-e8f11109d7a0-maas-api-tls\") pod \"maas-api-e2e-api-a-948a62-54b9f5d999-qtwbm\" (UID: \"085ec5e0-8366-47a6-abc1-e8f11109d7a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-948a62-54b9f5d999-qtwbm"
Jul 26 16:28:07.015476 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:07.015453    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jbz2\" (UniqueName: \"kubernetes.io/projected/085ec5e0-8366-47a6-abc1-e8f11109d7a0-kube-api-access-4jbz2\") pod \"maas-api-e2e-api-a-948a62-54b9f5d999-qtwbm\" (UID: \"085ec5e0-8366-47a6-abc1-e8f11109d7a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-948a62-54b9f5d999-qtwbm"
Jul 26 16:28:07.106158 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:07.106128    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-948a62-54b9f5d999-qtwbm"
Jul 26 16:28:07.255075 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:07.254999    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-948a62-54b9f5d999-qtwbm"]
Jul 26 16:28:07.258147 ip-10-0-133-204 kubenswrapper[2563]: W0726 16:28:07.258121    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod085ec5e0_8366_47a6_abc1_e8f11109d7a0.slice/crio-9e41c0e8df6a863c35791fc3da61654a163c3788bc4d34600f619f7ec0d09371 WatchSource:0}: Error finding container 9e41c0e8df6a863c35791fc3da61654a163c3788bc4d34600f619f7ec0d09371: Status 404 returned error can't find the container with id 9e41c0e8df6a863c35791fc3da61654a163c3788bc4d34600f619f7ec0d09371
Jul 26 16:28:07.259401 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:07.259386    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 26 16:28:07.398746 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:07.398712    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:28:07.484779 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:07.484733    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:28:07.489698 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:07.489657    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-948a62-5c776d57c6-phb2n"]
Jul 26 16:28:07.494836 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:07.494814    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-948a62-5c776d57c6-phb2n"
Jul 26 16:28:07.497484 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:07.497395    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-948a62\""
Jul 26 16:28:07.502536 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:07.502472    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-948a62-54b9f5d999-qtwbm" event={"ID":"085ec5e0-8366-47a6-abc1-e8f11109d7a0","Type":"ContainerStarted","Data":"9e41c0e8df6a863c35791fc3da61654a163c3788bc4d34600f619f7ec0d09371"}
Jul 26 16:28:07.503296 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:07.503271    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-948a62-5c776d57c6-phb2n"]
Jul 26 16:28:07.518829 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:07.516927    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-948a62-86df66c5cb-89dd7"]
Jul 26 16:28:07.524781 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:07.522081    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-948a62-86df66c5cb-89dd7"
Jul 26 16:28:07.530742 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:07.530267    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-948a62-dockercfg-pgn2t\""
Jul 26 16:28:07.536572 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:07.536543    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-948a62-86df66c5cb-89dd7"]
Jul 26 16:28:07.620086 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:07.620055    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7de7acb-8f26-4389-938c-47f60fa02531-plugins-config-volume\") pod \"payload-processing-e2e-api-a-948a62-5c776d57c6-phb2n\" (UID: \"e7de7acb-8f26-4389-938c-47f60fa02531\") " pod="openshift-ingress/payload-processing-e2e-api-a-948a62-5c776d57c6-phb2n"
Jul 26 16:28:07.620284 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:07.620190    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gws7\" (UniqueName: \"kubernetes.io/projected/d639acdd-8fd8-4f76-b135-d82c7d01cfc5-kube-api-access-7gws7\") pod \"payload-pre-processing-e2e-api-a-948a62-86df66c5cb-89dd7\" (UID: \"d639acdd-8fd8-4f76-b135-d82c7d01cfc5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-948a62-86df66c5cb-89dd7"
Jul 26 16:28:07.620284 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:07.620266    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d639acdd-8fd8-4f76-b135-d82c7d01cfc5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-948a62-86df66c5cb-89dd7\" (UID: \"d639acdd-8fd8-4f76-b135-d82c7d01cfc5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-948a62-86df66c5cb-89dd7"
Jul 26 16:28:07.620375 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:07.620297    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m757h\" (UniqueName: \"kubernetes.io/projected/e7de7acb-8f26-4389-938c-47f60fa02531-kube-api-access-m757h\") pod \"payload-processing-e2e-api-a-948a62-5c776d57c6-phb2n\" (UID: \"e7de7acb-8f26-4389-938c-47f60fa02531\") " pod="openshift-ingress/payload-processing-e2e-api-a-948a62-5c776d57c6-phb2n"
Jul 26 16:28:07.721856 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:07.721829    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gws7\" (UniqueName: \"kubernetes.io/projected/d639acdd-8fd8-4f76-b135-d82c7d01cfc5-kube-api-access-7gws7\") pod \"payload-pre-processing-e2e-api-a-948a62-86df66c5cb-89dd7\" (UID: \"d639acdd-8fd8-4f76-b135-d82c7d01cfc5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-948a62-86df66c5cb-89dd7"
Jul 26 16:28:07.722248 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:07.721874    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d639acdd-8fd8-4f76-b135-d82c7d01cfc5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-948a62-86df66c5cb-89dd7\" (UID: \"d639acdd-8fd8-4f76-b135-d82c7d01cfc5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-948a62-86df66c5cb-89dd7"
Jul 26 16:28:07.722248 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:07.721999    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m757h\" (UniqueName: \"kubernetes.io/projected/e7de7acb-8f26-4389-938c-47f60fa02531-kube-api-access-m757h\") pod \"payload-processing-e2e-api-a-948a62-5c776d57c6-phb2n\" (UID: \"e7de7acb-8f26-4389-938c-47f60fa02531\") " pod="openshift-ingress/payload-processing-e2e-api-a-948a62-5c776d57c6-phb2n"
Jul 26 16:28:07.722248 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:07.722069    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7de7acb-8f26-4389-938c-47f60fa02531-plugins-config-volume\") pod \"payload-processing-e2e-api-a-948a62-5c776d57c6-phb2n\" (UID: \"e7de7acb-8f26-4389-938c-47f60fa02531\") " pod="openshift-ingress/payload-processing-e2e-api-a-948a62-5c776d57c6-phb2n"
Jul 26 16:28:07.722704 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:07.722683    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d639acdd-8fd8-4f76-b135-d82c7d01cfc5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-948a62-86df66c5cb-89dd7\" (UID: \"d639acdd-8fd8-4f76-b135-d82c7d01cfc5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-948a62-86df66c5cb-89dd7"
Jul 26 16:28:07.722808 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:07.722746    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7de7acb-8f26-4389-938c-47f60fa02531-plugins-config-volume\") pod \"payload-processing-e2e-api-a-948a62-5c776d57c6-phb2n\" (UID: \"e7de7acb-8f26-4389-938c-47f60fa02531\") " pod="openshift-ingress/payload-processing-e2e-api-a-948a62-5c776d57c6-phb2n"
Jul 26 16:28:07.731328 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:07.731301    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m757h\" (UniqueName: \"kubernetes.io/projected/e7de7acb-8f26-4389-938c-47f60fa02531-kube-api-access-m757h\") pod \"payload-processing-e2e-api-a-948a62-5c776d57c6-phb2n\" (UID: \"e7de7acb-8f26-4389-938c-47f60fa02531\") " pod="openshift-ingress/payload-processing-e2e-api-a-948a62-5c776d57c6-phb2n"
Jul 26 16:28:07.731604 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:07.731588    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gws7\" (UniqueName: \"kubernetes.io/projected/d639acdd-8fd8-4f76-b135-d82c7d01cfc5-kube-api-access-7gws7\") pod \"payload-pre-processing-e2e-api-a-948a62-86df66c5cb-89dd7\" (UID: \"d639acdd-8fd8-4f76-b135-d82c7d01cfc5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-948a62-86df66c5cb-89dd7"
Jul 26 16:28:07.810737 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:07.810652    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-948a62-5c776d57c6-phb2n"
Jul 26 16:28:07.843582 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:07.843545    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-948a62-86df66c5cb-89dd7"
Jul 26 16:28:07.969488 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:07.969341    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-948a62-5c776d57c6-phb2n"]
Jul 26 16:28:07.999564 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:07.999541    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-948a62-86df66c5cb-89dd7"]
Jul 26 16:28:08.011922 ip-10-0-133-204 kubenswrapper[2563]: W0726 16:28:08.011888    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd639acdd_8fd8_4f76_b135_d82c7d01cfc5.slice/crio-1b20a428a851458b4363d323c694601a9dda82f9ca18e0166d106c0b3bbb6eec WatchSource:0}: Error finding container 1b20a428a851458b4363d323c694601a9dda82f9ca18e0166d106c0b3bbb6eec: Status 404 returned error can't find the container with id 1b20a428a851458b4363d323c694601a9dda82f9ca18e0166d106c0b3bbb6eec
Jul 26 16:28:08.510046 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:08.510005    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-948a62-54b9f5d999-qtwbm" event={"ID":"085ec5e0-8366-47a6-abc1-e8f11109d7a0","Type":"ContainerStarted","Data":"dd54b74ced47cad7e5ac9ed90f5adbdd2cf8eee7ef22bcfdd0cdfcd7c9faa8ea"}
Jul 26 16:28:08.510227 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:08.510132    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-948a62-54b9f5d999-qtwbm"
Jul 26 16:28:08.511631 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:08.511605    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-948a62-86df66c5cb-89dd7" event={"ID":"d639acdd-8fd8-4f76-b135-d82c7d01cfc5","Type":"ContainerStarted","Data":"27ef42f0ad760a8697448918c3edbb0d5c9750f7b313ce09cd5543af42c4c503"}
Jul 26 16:28:08.511756 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:08.511652    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-948a62-86df66c5cb-89dd7"
Jul 26 16:28:08.511756 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:08.511669    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-948a62-86df66c5cb-89dd7" event={"ID":"d639acdd-8fd8-4f76-b135-d82c7d01cfc5","Type":"ContainerStarted","Data":"1b20a428a851458b4363d323c694601a9dda82f9ca18e0166d106c0b3bbb6eec"}
Jul 26 16:28:08.513181 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:08.513158    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-948a62-5c776d57c6-phb2n" event={"ID":"e7de7acb-8f26-4389-938c-47f60fa02531","Type":"ContainerStarted","Data":"904c4ef382f26300ef00d84ac5447085b2ed952bcf94fc103cefdbae5840c354"}
Jul 26 16:28:08.513271 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:08.513197    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-948a62-5c776d57c6-phb2n" event={"ID":"e7de7acb-8f26-4389-938c-47f60fa02531","Type":"ContainerStarted","Data":"beb14cb6054deca3ddb119690966afe859144f7ebe97df45d0f6db15f4617756"}
Jul 26 16:28:08.513271 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:08.513233    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-948a62-5c776d57c6-phb2n"
Jul 26 16:28:08.528523 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:08.528465    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-948a62-54b9f5d999-qtwbm" podStartSLOduration=2.079175146 podStartE2EDuration="2.528448244s" totalImagesPullingTime="449.273098ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:28:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:28:08.528007123 +0000 UTC m=+2280.632232933" watchObservedRunningTime="2026-07-26 16:28:08.528448244 +0000 UTC m=+2280.632674056"
Jul 26 16:28:08.544984 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:08.544939    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-948a62-5c776d57c6-phb2n" podStartSLOduration=1.544923585 podStartE2EDuration="1.544923585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:28:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:28:08.543594913 +0000 UTC m=+2280.647820721" watchObservedRunningTime="2026-07-26 16:28:08.544923585 +0000 UTC m=+2280.649149394"
Jul 26 16:28:08.560157 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:08.560112    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-948a62-86df66c5cb-89dd7" podStartSLOduration=1.56009618 podStartE2EDuration="1.56009618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:28:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:28:08.559425534 +0000 UTC m=+2280.663651368" watchObservedRunningTime="2026-07-26 16:28:08.56009618 +0000 UTC m=+2280.664321989"
Jul 26 16:28:11.227948 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:11.227916    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:28:11.749057 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:11.749025    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:28:12.067269 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:12.067180    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:28:12.572098 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:12.572062    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:28:14.524770 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:14.524743    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-948a62-54b9f5d999-qtwbm"
Jul 26 16:28:14.753142 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:14.753099    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:28:14.856521 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:14.856426    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-df352b-78957f86bd-2ls7x"]
Jul 26 16:28:14.860366 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:14.860343    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-df352b-78957f86bd-2ls7x"
Jul 26 16:28:14.863343 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:14.863320    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-df352b-serving-cert\""
Jul 26 16:28:14.864400 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:14.864360    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:28:14.872642 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:14.872613    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-df352b-78957f86bd-2ls7x"]
Jul 26 16:28:14.950075 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:14.950041    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:28:14.992182 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:14.992148    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xzv7\" (UniqueName: \"kubernetes.io/projected/4cab2b78-d1c5-4598-b767-5e203100afa1-kube-api-access-7xzv7\") pod \"maas-api-e2e-api-b-df352b-78957f86bd-2ls7x\" (UID: \"4cab2b78-d1c5-4598-b767-5e203100afa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-df352b-78957f86bd-2ls7x"
Jul 26 16:28:14.992375 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:14.992297    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cab2b78-d1c5-4598-b767-5e203100afa1-maas-api-tls\") pod \"maas-api-e2e-api-b-df352b-78957f86bd-2ls7x\" (UID: \"4cab2b78-d1c5-4598-b767-5e203100afa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-df352b-78957f86bd-2ls7x"
Jul 26 16:28:14.992375 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:14.992346    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cab2b78-d1c5-4598-b767-5e203100afa1-openshift-service-ca\") pod \"maas-api-e2e-api-b-df352b-78957f86bd-2ls7x\" (UID: \"4cab2b78-d1c5-4598-b767-5e203100afa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-df352b-78957f86bd-2ls7x"
Jul 26 16:28:15.051717 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:15.051684    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:28:15.092966 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:15.092926    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cab2b78-d1c5-4598-b767-5e203100afa1-openshift-service-ca\") pod \"maas-api-e2e-api-b-df352b-78957f86bd-2ls7x\" (UID: \"4cab2b78-d1c5-4598-b767-5e203100afa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-df352b-78957f86bd-2ls7x"
Jul 26 16:28:15.093178 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:15.093095    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xzv7\" (UniqueName: \"kubernetes.io/projected/4cab2b78-d1c5-4598-b767-5e203100afa1-kube-api-access-7xzv7\") pod \"maas-api-e2e-api-b-df352b-78957f86bd-2ls7x\" (UID: \"4cab2b78-d1c5-4598-b767-5e203100afa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-df352b-78957f86bd-2ls7x"
Jul 26 16:28:15.093246 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:15.093201    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cab2b78-d1c5-4598-b767-5e203100afa1-maas-api-tls\") pod \"maas-api-e2e-api-b-df352b-78957f86bd-2ls7x\" (UID: \"4cab2b78-d1c5-4598-b767-5e203100afa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-df352b-78957f86bd-2ls7x"
Jul 26 16:28:15.093650 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:15.093629    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cab2b78-d1c5-4598-b767-5e203100afa1-openshift-service-ca\") pod \"maas-api-e2e-api-b-df352b-78957f86bd-2ls7x\" (UID: \"4cab2b78-d1c5-4598-b767-5e203100afa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-df352b-78957f86bd-2ls7x"
Jul 26 16:28:15.095706 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:15.095680    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cab2b78-d1c5-4598-b767-5e203100afa1-maas-api-tls\") pod \"maas-api-e2e-api-b-df352b-78957f86bd-2ls7x\" (UID: \"4cab2b78-d1c5-4598-b767-5e203100afa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-df352b-78957f86bd-2ls7x"
Jul 26 16:28:15.100688 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:15.100662    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xzv7\" (UniqueName: \"kubernetes.io/projected/4cab2b78-d1c5-4598-b767-5e203100afa1-kube-api-access-7xzv7\") pod \"maas-api-e2e-api-b-df352b-78957f86bd-2ls7x\" (UID: \"4cab2b78-d1c5-4598-b767-5e203100afa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-df352b-78957f86bd-2ls7x"
Jul 26 16:28:15.174827 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:15.174710    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-df352b-78957f86bd-2ls7x"
Jul 26 16:28:15.313236 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:15.313212    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-df352b-78957f86bd-2ls7x"]
Jul 26 16:28:15.315951 ip-10-0-133-204 kubenswrapper[2563]: W0726 16:28:15.315925    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cab2b78_d1c5_4598_b767_5e203100afa1.slice/crio-9a390b9aaa4546a254c16aba3e4d00b6b4fcc13dd6310b57b62dd94ec82c5ecb WatchSource:0}: Error finding container 9a390b9aaa4546a254c16aba3e4d00b6b4fcc13dd6310b57b62dd94ec82c5ecb: Status 404 returned error can't find the container with id 9a390b9aaa4546a254c16aba3e4d00b6b4fcc13dd6310b57b62dd94ec82c5ecb
Jul 26 16:28:15.546729 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:15.546695    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:28:15.548338 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:15.548315    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-df352b-78957f86bd-2ls7x" event={"ID":"4cab2b78-d1c5-4598-b767-5e203100afa1","Type":"ContainerStarted","Data":"9a390b9aaa4546a254c16aba3e4d00b6b4fcc13dd6310b57b62dd94ec82c5ecb"}
Jul 26 16:28:15.640224 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:15.640189    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:28:15.682436 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:15.682401    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-df352b-676ffff8fd-q8kqn"]
Jul 26 16:28:15.697304 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:15.696208    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-df352b-676ffff8fd-q8kqn"
Jul 26 16:28:15.699545 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:15.699062    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-df352b-dockercfg-tkrnh\""
Jul 26 16:28:15.699545 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:15.699075    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-df352b\""
Jul 26 16:28:15.700175 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:15.700148    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-df352b-88459dd-9t7bq"]
Jul 26 16:28:15.705472 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:15.705281    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-df352b-676ffff8fd-q8kqn"]
Jul 26 16:28:15.705588 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:15.705402    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-df352b-88459dd-9t7bq"
Jul 26 16:28:15.712144 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:15.712085    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-df352b-88459dd-9t7bq"]
Jul 26 16:28:15.800683 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:15.800589    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6wll\" (UniqueName: \"kubernetes.io/projected/d50bb5a9-4d7b-4053-b035-0e554ef18d0c-kube-api-access-t6wll\") pod \"payload-pre-processing-e2e-api-b-df352b-88459dd-9t7bq\" (UID: \"d50bb5a9-4d7b-4053-b035-0e554ef18d0c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-df352b-88459dd-9t7bq"
Jul 26 16:28:15.800683 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:15.800632    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d50bb5a9-4d7b-4053-b035-0e554ef18d0c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-df352b-88459dd-9t7bq\" (UID: \"d50bb5a9-4d7b-4053-b035-0e554ef18d0c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-df352b-88459dd-9t7bq"
Jul 26 16:28:15.800683 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:15.800681    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb8ba3f6-8d9b-4478-a127-a9802ef8ca7e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-df352b-676ffff8fd-q8kqn\" (UID: \"eb8ba3f6-8d9b-4478-a127-a9802ef8ca7e\") " pod="openshift-ingress/payload-processing-e2e-api-b-df352b-676ffff8fd-q8kqn"
Jul 26 16:28:15.800997 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:15.800754    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gch4p\" (UniqueName: \"kubernetes.io/projected/eb8ba3f6-8d9b-4478-a127-a9802ef8ca7e-kube-api-access-gch4p\") pod \"payload-processing-e2e-api-b-df352b-676ffff8fd-q8kqn\" (UID: \"eb8ba3f6-8d9b-4478-a127-a9802ef8ca7e\") " pod="openshift-ingress/payload-processing-e2e-api-b-df352b-676ffff8fd-q8kqn"
Jul 26 16:28:15.902286 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:15.902247    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gch4p\" (UniqueName: \"kubernetes.io/projected/eb8ba3f6-8d9b-4478-a127-a9802ef8ca7e-kube-api-access-gch4p\") pod \"payload-processing-e2e-api-b-df352b-676ffff8fd-q8kqn\" (UID: \"eb8ba3f6-8d9b-4478-a127-a9802ef8ca7e\") " pod="openshift-ingress/payload-processing-e2e-api-b-df352b-676ffff8fd-q8kqn"
Jul 26 16:28:15.902479 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:15.902361    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6wll\" (UniqueName: \"kubernetes.io/projected/d50bb5a9-4d7b-4053-b035-0e554ef18d0c-kube-api-access-t6wll\") pod \"payload-pre-processing-e2e-api-b-df352b-88459dd-9t7bq\" (UID: \"d50bb5a9-4d7b-4053-b035-0e554ef18d0c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-df352b-88459dd-9t7bq"
Jul 26 16:28:15.902479 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:15.902408    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d50bb5a9-4d7b-4053-b035-0e554ef18d0c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-df352b-88459dd-9t7bq\" (UID: \"d50bb5a9-4d7b-4053-b035-0e554ef18d0c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-df352b-88459dd-9t7bq"
Jul 26 16:28:15.902479 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:15.902455    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb8ba3f6-8d9b-4478-a127-a9802ef8ca7e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-df352b-676ffff8fd-q8kqn\" (UID: \"eb8ba3f6-8d9b-4478-a127-a9802ef8ca7e\") " pod="openshift-ingress/payload-processing-e2e-api-b-df352b-676ffff8fd-q8kqn"
Jul 26 16:28:15.903394 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:15.903367    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb8ba3f6-8d9b-4478-a127-a9802ef8ca7e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-df352b-676ffff8fd-q8kqn\" (UID: \"eb8ba3f6-8d9b-4478-a127-a9802ef8ca7e\") " pod="openshift-ingress/payload-processing-e2e-api-b-df352b-676ffff8fd-q8kqn"
Jul 26 16:28:15.903617 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:15.903433    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d50bb5a9-4d7b-4053-b035-0e554ef18d0c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-df352b-88459dd-9t7bq\" (UID: \"d50bb5a9-4d7b-4053-b035-0e554ef18d0c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-df352b-88459dd-9t7bq"
Jul 26 16:28:15.910885 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:15.910858    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gch4p\" (UniqueName: \"kubernetes.io/projected/eb8ba3f6-8d9b-4478-a127-a9802ef8ca7e-kube-api-access-gch4p\") pod \"payload-processing-e2e-api-b-df352b-676ffff8fd-q8kqn\" (UID: \"eb8ba3f6-8d9b-4478-a127-a9802ef8ca7e\") " pod="openshift-ingress/payload-processing-e2e-api-b-df352b-676ffff8fd-q8kqn"
Jul 26 16:28:15.910997 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:15.910886    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6wll\" (UniqueName: \"kubernetes.io/projected/d50bb5a9-4d7b-4053-b035-0e554ef18d0c-kube-api-access-t6wll\") pod \"payload-pre-processing-e2e-api-b-df352b-88459dd-9t7bq\" (UID: \"d50bb5a9-4d7b-4053-b035-0e554ef18d0c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-df352b-88459dd-9t7bq"
Jul 26 16:28:16.019688 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:16.019655    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-df352b-676ffff8fd-q8kqn"
Jul 26 16:28:16.040062 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:16.040030    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-df352b-88459dd-9t7bq"
Jul 26 16:28:16.211842 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:16.211772    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-df352b-88459dd-9t7bq"]
Jul 26 16:28:16.214449 ip-10-0-133-204 kubenswrapper[2563]: W0726 16:28:16.214397    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd50bb5a9_4d7b_4053_b035_0e554ef18d0c.slice/crio-8d52281872e8bc8c464c8569e25ac130b19001261da1b9912ba9b8740e110704 WatchSource:0}: Error finding container 8d52281872e8bc8c464c8569e25ac130b19001261da1b9912ba9b8740e110704: Status 404 returned error can't find the container with id 8d52281872e8bc8c464c8569e25ac130b19001261da1b9912ba9b8740e110704
Jul 26 16:28:16.384884 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:16.384857    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-df352b-676ffff8fd-q8kqn"]
Jul 26 16:28:16.386121 ip-10-0-133-204 kubenswrapper[2563]: W0726 16:28:16.386091    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb8ba3f6_8d9b_4478_a127_a9802ef8ca7e.slice/crio-dca84e57b861568775762be4c1d01e10211ec0bfc8aad419cfe8c1d01a2233b4 WatchSource:0}: Error finding container dca84e57b861568775762be4c1d01e10211ec0bfc8aad419cfe8c1d01a2233b4: Status 404 returned error can't find the container with id dca84e57b861568775762be4c1d01e10211ec0bfc8aad419cfe8c1d01a2233b4
Jul 26 16:28:16.554189 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:16.554145    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-df352b-88459dd-9t7bq" event={"ID":"d50bb5a9-4d7b-4053-b035-0e554ef18d0c","Type":"ContainerStarted","Data":"f9d189ef880c0ef9d24a9d8b4feba790f9f749f561e16a739706784ba8d7c057"}
Jul 26 16:28:16.554656 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:16.554206    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-df352b-88459dd-9t7bq" event={"ID":"d50bb5a9-4d7b-4053-b035-0e554ef18d0c","Type":"ContainerStarted","Data":"8d52281872e8bc8c464c8569e25ac130b19001261da1b9912ba9b8740e110704"}
Jul 26 16:28:16.554656 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:16.554255    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-df352b-88459dd-9t7bq"
Jul 26 16:28:16.555564 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:16.555539    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-df352b-676ffff8fd-q8kqn" event={"ID":"eb8ba3f6-8d9b-4478-a127-a9802ef8ca7e","Type":"ContainerStarted","Data":"485b512976b25ae240dac8cbe8c06e3ee78c0dee4ec0c2c2dbcfc776e19fe861"}
Jul 26 16:28:16.555680 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:16.555589    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-df352b-676ffff8fd-q8kqn" event={"ID":"eb8ba3f6-8d9b-4478-a127-a9802ef8ca7e","Type":"ContainerStarted","Data":"dca84e57b861568775762be4c1d01e10211ec0bfc8aad419cfe8c1d01a2233b4"}
Jul 26 16:28:16.555680 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:16.555620    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-df352b-676ffff8fd-q8kqn"
Jul 26 16:28:16.557252 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:16.557228    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-df352b-78957f86bd-2ls7x" event={"ID":"4cab2b78-d1c5-4598-b767-5e203100afa1","Type":"ContainerStarted","Data":"11181dc01e7f98715ccb8d5de85138bc841fe443b0fa574fb2d25227b639e49e"}
Jul 26 16:28:16.557354 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:16.557280    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-df352b-78957f86bd-2ls7x"
Jul 26 16:28:16.571917 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:16.571873    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-df352b-88459dd-9t7bq" podStartSLOduration=1.5718575339999998 podStartE2EDuration="1.571857534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:28:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:28:16.57079779 +0000 UTC m=+2288.675023600" watchObservedRunningTime="2026-07-26 16:28:16.571857534 +0000 UTC m=+2288.676083342"
Jul 26 16:28:16.589171 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:16.589119    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-df352b-78957f86bd-2ls7x" podStartSLOduration=2.211537925 podStartE2EDuration="2.589100576s" totalImagesPullingTime="377.562651ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:28:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:28:16.585822419 +0000 UTC m=+2288.690048252" watchObservedRunningTime="2026-07-26 16:28:16.589100576 +0000 UTC m=+2288.693326387"
Jul 26 16:28:16.605149 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:16.605104    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-df352b-676ffff8fd-q8kqn" podStartSLOduration=1.605090241 podStartE2EDuration="1.605090241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:28:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:28:16.602069407 +0000 UTC m=+2288.706295216" watchObservedRunningTime="2026-07-26 16:28:16.605090241 +0000 UTC m=+2288.709316090"
Jul 26 16:28:19.266155 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:19.266117    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:28:19.520743 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:19.520667    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-948a62-86df66c5cb-89dd7"
Jul 26 16:28:19.520913 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:19.520747    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-948a62-5c776d57c6-phb2n"
Jul 26 16:28:22.569089 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:22.569060    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-df352b-78957f86bd-2ls7x"
Jul 26 16:28:27.565262 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:27.565234    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-df352b-676ffff8fd-q8kqn"
Jul 26 16:28:27.565861 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:27.565320    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-df352b-88459dd-9t7bq"
Jul 26 16:28:43.248811 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:43.248778    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-948a62-54b9f5d999-qtwbm"]
Jul 26 16:28:43.249223 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:43.249021    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-948a62-54b9f5d999-qtwbm" podUID="085ec5e0-8366-47a6-abc1-e8f11109d7a0" containerName="maas-api" containerID="cri-o://dd54b74ced47cad7e5ac9ed90f5adbdd2cf8eee7ef22bcfdd0cdfcd7c9faa8ea" gracePeriod=30
Jul 26 16:28:43.280805 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:43.280743    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:28:43.314662 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:43.314631    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-948a62-5c776d57c6-phb2n"]
Jul 26 16:28:43.314890 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:43.314862    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-948a62-5c776d57c6-phb2n" podUID="e7de7acb-8f26-4389-938c-47f60fa02531" containerName="payload-processing" containerID="cri-o://904c4ef382f26300ef00d84ac5447085b2ed952bcf94fc103cefdbae5840c354" gracePeriod=30
Jul 26 16:28:43.329925 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:43.329895    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-948a62-86df66c5cb-89dd7"]
Jul 26 16:28:43.330097 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:43.330075    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-948a62-86df66c5cb-89dd7" podUID="d639acdd-8fd8-4f76-b135-d82c7d01cfc5" containerName="payload-pre-processing" containerID="cri-o://27ef42f0ad760a8697448918c3edbb0d5c9750f7b313ce09cd5543af42c4c503" gracePeriod=30
Jul 26 16:28:43.424294 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:43.424259    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:28:43.537166 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:43.537123    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:28:43.612449 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:43.612423    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-948a62-54b9f5d999-qtwbm"
Jul 26 16:28:43.645402 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:43.645379    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-948a62-5c776d57c6-phb2n"
Jul 26 16:28:43.648577 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:43.648555    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-948a62-86df66c5cb-89dd7"
Jul 26 16:28:43.675170 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:43.675133    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/085ec5e0-8366-47a6-abc1-e8f11109d7a0-openshift-service-ca\") pod \"085ec5e0-8366-47a6-abc1-e8f11109d7a0\" (UID: \"085ec5e0-8366-47a6-abc1-e8f11109d7a0\") "
Jul 26 16:28:43.675367 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:43.675180    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4jbz2\" (UniqueName: \"kubernetes.io/projected/085ec5e0-8366-47a6-abc1-e8f11109d7a0-kube-api-access-4jbz2\") pod \"085ec5e0-8366-47a6-abc1-e8f11109d7a0\" (UID: \"085ec5e0-8366-47a6-abc1-e8f11109d7a0\") "
Jul 26 16:28:43.675367 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:43.675208    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/085ec5e0-8366-47a6-abc1-e8f11109d7a0-maas-api-tls\") pod \"085ec5e0-8366-47a6-abc1-e8f11109d7a0\" (UID: \"085ec5e0-8366-47a6-abc1-e8f11109d7a0\") "
Jul 26 16:28:43.675568 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:43.675542    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/085ec5e0-8366-47a6-abc1-e8f11109d7a0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "085ec5e0-8366-47a6-abc1-e8f11109d7a0" (UID: "085ec5e0-8366-47a6-abc1-e8f11109d7a0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 26 16:28:43.677380 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:43.677353    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/085ec5e0-8366-47a6-abc1-e8f11109d7a0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "085ec5e0-8366-47a6-abc1-e8f11109d7a0" (UID: "085ec5e0-8366-47a6-abc1-e8f11109d7a0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 26 16:28:43.677581 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:43.677561    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/085ec5e0-8366-47a6-abc1-e8f11109d7a0-kube-api-access-4jbz2" (OuterVolumeSpecName: "kube-api-access-4jbz2") pod "085ec5e0-8366-47a6-abc1-e8f11109d7a0" (UID: "085ec5e0-8366-47a6-abc1-e8f11109d7a0"). InnerVolumeSpecName "kube-api-access-4jbz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 26 16:28:43.696232 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:43.696169    2563 generic.go:356] "Generic (PLEG): container finished" podID="085ec5e0-8366-47a6-abc1-e8f11109d7a0" containerID="dd54b74ced47cad7e5ac9ed90f5adbdd2cf8eee7ef22bcfdd0cdfcd7c9faa8ea" exitCode=0
Jul 26 16:28:43.696232 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:43.696225    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-948a62-54b9f5d999-qtwbm"
Jul 26 16:28:43.696509 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:43.696264    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-948a62-54b9f5d999-qtwbm" event={"ID":"085ec5e0-8366-47a6-abc1-e8f11109d7a0","Type":"ContainerDied","Data":"dd54b74ced47cad7e5ac9ed90f5adbdd2cf8eee7ef22bcfdd0cdfcd7c9faa8ea"}
Jul 26 16:28:43.696509 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:43.696446    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-948a62-54b9f5d999-qtwbm" event={"ID":"085ec5e0-8366-47a6-abc1-e8f11109d7a0","Type":"ContainerDied","Data":"9e41c0e8df6a863c35791fc3da61654a163c3788bc4d34600f619f7ec0d09371"}
Jul 26 16:28:43.696509 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:43.696507    2563 scope.go:117] "RemoveContainer" containerID="dd54b74ced47cad7e5ac9ed90f5adbdd2cf8eee7ef22bcfdd0cdfcd7c9faa8ea"
Jul 26 16:28:43.698317 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:43.698193    2563 generic.go:356] "Generic (PLEG): container finished" podID="d639acdd-8fd8-4f76-b135-d82c7d01cfc5" containerID="27ef42f0ad760a8697448918c3edbb0d5c9750f7b313ce09cd5543af42c4c503" exitCode=0
Jul 26 16:28:43.698317 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:43.698244    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-948a62-86df66c5cb-89dd7"
Jul 26 16:28:43.698317 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:43.698287    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-948a62-86df66c5cb-89dd7" event={"ID":"d639acdd-8fd8-4f76-b135-d82c7d01cfc5","Type":"ContainerDied","Data":"27ef42f0ad760a8697448918c3edbb0d5c9750f7b313ce09cd5543af42c4c503"}
Jul 26 16:28:43.698558 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:43.698338    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-948a62-86df66c5cb-89dd7" event={"ID":"d639acdd-8fd8-4f76-b135-d82c7d01cfc5","Type":"ContainerDied","Data":"1b20a428a851458b4363d323c694601a9dda82f9ca18e0166d106c0b3bbb6eec"}
Jul 26 16:28:43.699931 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:43.699748    2563 generic.go:356] "Generic (PLEG): container finished" podID="e7de7acb-8f26-4389-938c-47f60fa02531" containerID="904c4ef382f26300ef00d84ac5447085b2ed952bcf94fc103cefdbae5840c354" exitCode=0
Jul 26 16:28:43.699931 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:43.699823    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-948a62-5c776d57c6-phb2n"
Jul 26 16:28:43.699931 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:43.699862    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-948a62-5c776d57c6-phb2n" event={"ID":"e7de7acb-8f26-4389-938c-47f60fa02531","Type":"ContainerDied","Data":"904c4ef382f26300ef00d84ac5447085b2ed952bcf94fc103cefdbae5840c354"}
Jul 26 16:28:43.699931 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:43.699899    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-948a62-5c776d57c6-phb2n" event={"ID":"e7de7acb-8f26-4389-938c-47f60fa02531","Type":"ContainerDied","Data":"beb14cb6054deca3ddb119690966afe859144f7ebe97df45d0f6db15f4617756"}
Jul 26 16:28:43.706977 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:43.706938    2563 scope.go:117] "RemoveContainer" containerID="dd54b74ced47cad7e5ac9ed90f5adbdd2cf8eee7ef22bcfdd0cdfcd7c9faa8ea"
Jul 26 16:28:43.707261 ip-10-0-133-204 kubenswrapper[2563]: E0726 16:28:43.707240    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd54b74ced47cad7e5ac9ed90f5adbdd2cf8eee7ef22bcfdd0cdfcd7c9faa8ea\": container with ID starting with dd54b74ced47cad7e5ac9ed90f5adbdd2cf8eee7ef22bcfdd0cdfcd7c9faa8ea not found: ID does not exist" containerID="dd54b74ced47cad7e5ac9ed90f5adbdd2cf8eee7ef22bcfdd0cdfcd7c9faa8ea"
Jul 26 16:28:43.707345 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:43.707274    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd54b74ced47cad7e5ac9ed90f5adbdd2cf8eee7ef22bcfdd0cdfcd7c9faa8ea"} err="failed to get container status \"dd54b74ced47cad7e5ac9ed90f5adbdd2cf8eee7ef22bcfdd0cdfcd7c9faa8ea\": rpc error: code = NotFound desc = could not find container \"dd54b74ced47cad7e5ac9ed90f5adbdd2cf8eee7ef22bcfdd0cdfcd7c9faa8ea\": container with ID starting with dd54b74ced47cad7e5ac9ed90f5adbdd2cf8eee7ef22bcfdd0cdfcd7c9faa8ea not found: ID does not exist"
Jul 26 16:28:43.707345 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:43.707315    2563 scope.go:117] "RemoveContainer" containerID="27ef42f0ad760a8697448918c3edbb0d5c9750f7b313ce09cd5543af42c4c503"
Jul 26 16:28:43.717371 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:43.717349    2563 scope.go:117] "RemoveContainer" containerID="27ef42f0ad760a8697448918c3edbb0d5c9750f7b313ce09cd5543af42c4c503"
Jul 26 16:28:43.717629 ip-10-0-133-204 kubenswrapper[2563]: E0726 16:28:43.717611    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27ef42f0ad760a8697448918c3edbb0d5c9750f7b313ce09cd5543af42c4c503\": container with ID starting with 27ef42f0ad760a8697448918c3edbb0d5c9750f7b313ce09cd5543af42c4c503 not found: ID does not exist" containerID="27ef42f0ad760a8697448918c3edbb0d5c9750f7b313ce09cd5543af42c4c503"
Jul 26 16:28:43.717699 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:43.717642    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27ef42f0ad760a8697448918c3edbb0d5c9750f7b313ce09cd5543af42c4c503"} err="failed to get container status \"27ef42f0ad760a8697448918c3edbb0d5c9750f7b313ce09cd5543af42c4c503\": rpc error: code = NotFound desc = could not find container \"27ef42f0ad760a8697448918c3edbb0d5c9750f7b313ce09cd5543af42c4c503\": container with ID starting with 27ef42f0ad760a8697448918c3edbb0d5c9750f7b313ce09cd5543af42c4c503 not found: ID does not exist"
Jul 26 16:28:43.717699 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:43.717668    2563 scope.go:117] "RemoveContainer" containerID="904c4ef382f26300ef00d84ac5447085b2ed952bcf94fc103cefdbae5840c354"
Jul 26 16:28:43.726634 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:43.726616    2563 scope.go:117] "RemoveContainer" containerID="904c4ef382f26300ef00d84ac5447085b2ed952bcf94fc103cefdbae5840c354"
Jul 26 16:28:43.726902 ip-10-0-133-204 kubenswrapper[2563]: E0726 16:28:43.726885    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"904c4ef382f26300ef00d84ac5447085b2ed952bcf94fc103cefdbae5840c354\": container with ID starting with 904c4ef382f26300ef00d84ac5447085b2ed952bcf94fc103cefdbae5840c354 not found: ID does not exist" containerID="904c4ef382f26300ef00d84ac5447085b2ed952bcf94fc103cefdbae5840c354"
Jul 26 16:28:43.726952 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:43.726908    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"904c4ef382f26300ef00d84ac5447085b2ed952bcf94fc103cefdbae5840c354"} err="failed to get container status \"904c4ef382f26300ef00d84ac5447085b2ed952bcf94fc103cefdbae5840c354\": rpc error: code = NotFound desc = could not find container \"904c4ef382f26300ef00d84ac5447085b2ed952bcf94fc103cefdbae5840c354\": container with ID starting with 904c4ef382f26300ef00d84ac5447085b2ed952bcf94fc103cefdbae5840c354 not found: ID does not exist"
Jul 26 16:28:43.727206 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:43.727188    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-948a62-54b9f5d999-qtwbm"]
Jul 26 16:28:43.732663 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:43.732638    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-948a62-54b9f5d999-qtwbm"]
Jul 26 16:28:43.776347 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:43.776260    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7gws7\" (UniqueName: \"kubernetes.io/projected/d639acdd-8fd8-4f76-b135-d82c7d01cfc5-kube-api-access-7gws7\") pod \"d639acdd-8fd8-4f76-b135-d82c7d01cfc5\" (UID: \"d639acdd-8fd8-4f76-b135-d82c7d01cfc5\") "
Jul 26 16:28:43.776347 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:43.776336    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7de7acb-8f26-4389-938c-47f60fa02531-plugins-config-volume\") pod \"e7de7acb-8f26-4389-938c-47f60fa02531\" (UID: \"e7de7acb-8f26-4389-938c-47f60fa02531\") "
Jul 26 16:28:43.776601 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:43.776365    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d639acdd-8fd8-4f76-b135-d82c7d01cfc5-plugins-config-volume\") pod \"d639acdd-8fd8-4f76-b135-d82c7d01cfc5\" (UID: \"d639acdd-8fd8-4f76-b135-d82c7d01cfc5\") "
Jul 26 16:28:43.776601 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:43.776428    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m757h\" (UniqueName: \"kubernetes.io/projected/e7de7acb-8f26-4389-938c-47f60fa02531-kube-api-access-m757h\") pod \"e7de7acb-8f26-4389-938c-47f60fa02531\" (UID: \"e7de7acb-8f26-4389-938c-47f60fa02531\") "
Jul 26 16:28:43.776775 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:43.776734    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/085ec5e0-8366-47a6-abc1-e8f11109d7a0-openshift-service-ca\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:28:43.776899 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:43.776754    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jbz2\" (UniqueName: \"kubernetes.io/projected/085ec5e0-8366-47a6-abc1-e8f11109d7a0-kube-api-access-4jbz2\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:28:43.776899 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:43.776797    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/085ec5e0-8366-47a6-abc1-e8f11109d7a0-maas-api-tls\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:28:43.776899 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:43.776807    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7de7acb-8f26-4389-938c-47f60fa02531-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e7de7acb-8f26-4389-938c-47f60fa02531" (UID: "e7de7acb-8f26-4389-938c-47f60fa02531"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 26 16:28:43.776899 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:43.776811    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d639acdd-8fd8-4f76-b135-d82c7d01cfc5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d639acdd-8fd8-4f76-b135-d82c7d01cfc5" (UID: "d639acdd-8fd8-4f76-b135-d82c7d01cfc5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 26 16:28:43.778517 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:43.778496    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d639acdd-8fd8-4f76-b135-d82c7d01cfc5-kube-api-access-7gws7" (OuterVolumeSpecName: "kube-api-access-7gws7") pod "d639acdd-8fd8-4f76-b135-d82c7d01cfc5" (UID: "d639acdd-8fd8-4f76-b135-d82c7d01cfc5"). InnerVolumeSpecName "kube-api-access-7gws7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 26 16:28:43.778598 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:43.778526    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7de7acb-8f26-4389-938c-47f60fa02531-kube-api-access-m757h" (OuterVolumeSpecName: "kube-api-access-m757h") pod "e7de7acb-8f26-4389-938c-47f60fa02531" (UID: "e7de7acb-8f26-4389-938c-47f60fa02531"). InnerVolumeSpecName "kube-api-access-m757h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 26 16:28:43.877442 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:43.877405    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gws7\" (UniqueName: \"kubernetes.io/projected/d639acdd-8fd8-4f76-b135-d82c7d01cfc5-kube-api-access-7gws7\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:28:43.877442 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:43.877439    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7de7acb-8f26-4389-938c-47f60fa02531-plugins-config-volume\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:28:43.877442 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:43.877451    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d639acdd-8fd8-4f76-b135-d82c7d01cfc5-plugins-config-volume\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:28:43.877680 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:43.877464    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m757h\" (UniqueName: \"kubernetes.io/projected/e7de7acb-8f26-4389-938c-47f60fa02531-kube-api-access-m757h\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:28:44.023006 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:44.022975    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-948a62-5c776d57c6-phb2n"]
Jul 26 16:28:44.027093 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:44.027018    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-948a62-5c776d57c6-phb2n"]
Jul 26 16:28:44.038923 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:44.038893    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-948a62-86df66c5cb-89dd7"]
Jul 26 16:28:44.043974 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:44.043947    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-948a62-86df66c5cb-89dd7"]
Jul 26 16:28:44.374111 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:44.374007    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="085ec5e0-8366-47a6-abc1-e8f11109d7a0" path="/var/lib/kubelet/pods/085ec5e0-8366-47a6-abc1-e8f11109d7a0/volumes"
Jul 26 16:28:44.374555 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:44.374383    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d639acdd-8fd8-4f76-b135-d82c7d01cfc5" path="/var/lib/kubelet/pods/d639acdd-8fd8-4f76-b135-d82c7d01cfc5/volumes"
Jul 26 16:28:44.374695 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:44.374681    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7de7acb-8f26-4389-938c-47f60fa02531" path="/var/lib/kubelet/pods/e7de7acb-8f26-4389-938c-47f60fa02531/volumes"
Jul 26 16:28:47.536369 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:47.536333    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:28:47.929238 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:47.929149    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:28:53.650139 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:53.650100    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:28:53.741533 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:53.741501    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:28:53.924257 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:53.924171    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:28:54.166161 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:54.166133    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:28:54.284421 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:54.284385    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-df352b-jtxrv"]
Jul 26 16:28:54.284910 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:54.284892    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="085ec5e0-8366-47a6-abc1-e8f11109d7a0" containerName="maas-api"
Jul 26 16:28:54.285022 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:54.284911    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="085ec5e0-8366-47a6-abc1-e8f11109d7a0" containerName="maas-api"
Jul 26 16:28:54.285022 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:54.284930    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d639acdd-8fd8-4f76-b135-d82c7d01cfc5" containerName="payload-pre-processing"
Jul 26 16:28:54.285022 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:54.284938    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d639acdd-8fd8-4f76-b135-d82c7d01cfc5" containerName="payload-pre-processing"
Jul 26 16:28:54.285022 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:54.284955    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7de7acb-8f26-4389-938c-47f60fa02531" containerName="payload-processing"
Jul 26 16:28:54.285022 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:54.284964    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7de7acb-8f26-4389-938c-47f60fa02531" containerName="payload-processing"
Jul 26 16:28:54.285262 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:54.285079    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d639acdd-8fd8-4f76-b135-d82c7d01cfc5" containerName="payload-pre-processing"
Jul 26 16:28:54.285262 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:54.285096    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="085ec5e0-8366-47a6-abc1-e8f11109d7a0" containerName="maas-api"
Jul 26 16:28:54.285262 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:54.285106    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7de7acb-8f26-4389-938c-47f60fa02531" containerName="payload-processing"
Jul 26 16:28:54.289827 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:54.289805    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-df352b-jtxrv"
Jul 26 16:28:54.292031 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:54.292012    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlwwk\""
Jul 26 16:28:54.296142 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:54.296118    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-df352b-jtxrv"]
Jul 26 16:28:54.381862 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:54.381833    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec2cbe93-a640-4bdb-b942-b17273a3b89e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-df352b-jtxrv\" (UID: \"ec2cbe93-a640-4bdb-b942-b17273a3b89e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-df352b-jtxrv"
Jul 26 16:28:54.482346 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:54.482306    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec2cbe93-a640-4bdb-b942-b17273a3b89e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-df352b-jtxrv\" (UID: \"ec2cbe93-a640-4bdb-b942-b17273a3b89e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-df352b-jtxrv"
Jul 26 16:28:54.482996 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:54.482975    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec2cbe93-a640-4bdb-b942-b17273a3b89e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-df352b-jtxrv\" (UID: \"ec2cbe93-a640-4bdb-b942-b17273a3b89e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-df352b-jtxrv"
Jul 26 16:28:54.602702 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:54.602618    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-df352b-jtxrv"
Jul 26 16:28:54.746797 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:54.746613    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-df352b-jtxrv"]
Jul 26 16:28:54.749174 ip-10-0-133-204 kubenswrapper[2563]: W0726 16:28:54.749128    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec2cbe93_a640_4bdb_b942_b17273a3b89e.slice/crio-4cf056ab02d5f418dcacb66a173b849d357609bdae8e790524c077b82a7c6c72 WatchSource:0}: Error finding container 4cf056ab02d5f418dcacb66a173b849d357609bdae8e790524c077b82a7c6c72: Status 404 returned error can't find the container with id 4cf056ab02d5f418dcacb66a173b849d357609bdae8e790524c077b82a7c6c72
Jul 26 16:28:54.761636 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:54.761596    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-df352b-jtxrv" event={"ID":"ec2cbe93-a640-4bdb-b942-b17273a3b89e","Type":"ContainerStarted","Data":"4cf056ab02d5f418dcacb66a173b849d357609bdae8e790524c077b82a7c6c72"}
Jul 26 16:28:57.780392 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:57.780353    2563 generic.go:356] "Generic (PLEG): container finished" podID="ec2cbe93-a640-4bdb-b942-b17273a3b89e" containerID="3aeabdb0012b605de8815d5b40b53b4b74989fdf33935fe1427a557b0a9eb601" exitCode=0
Jul 26 16:28:57.780870 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:57.780425    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-df352b-jtxrv" event={"ID":"ec2cbe93-a640-4bdb-b942-b17273a3b89e","Type":"ContainerDied","Data":"3aeabdb0012b605de8815d5b40b53b4b74989fdf33935fe1427a557b0a9eb601"}
Jul 26 16:28:58.919289 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:58.919264    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-df352b-jtxrv"
Jul 26 16:28:59.028085 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:59.028052    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec2cbe93-a640-4bdb-b942-b17273a3b89e-maas-api-service-ca\") pod \"ec2cbe93-a640-4bdb-b942-b17273a3b89e\" (UID: \"ec2cbe93-a640-4bdb-b942-b17273a3b89e\") "
Jul 26 16:28:59.028418 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:59.028394    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec2cbe93-a640-4bdb-b942-b17273a3b89e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ec2cbe93-a640-4bdb-b942-b17273a3b89e" (UID: "ec2cbe93-a640-4bdb-b942-b17273a3b89e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 26 16:28:59.129136 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:59.129047    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec2cbe93-a640-4bdb-b942-b17273a3b89e-maas-api-service-ca\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:28:59.790455 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:59.790427    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-df352b-jtxrv"
Jul 26 16:28:59.790632 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:59.790460    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-df352b-jtxrv" event={"ID":"ec2cbe93-a640-4bdb-b942-b17273a3b89e","Type":"ContainerDied","Data":"4cf056ab02d5f418dcacb66a173b849d357609bdae8e790524c077b82a7c6c72"}
Jul 26 16:28:59.790632 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:28:59.790501    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4cf056ab02d5f418dcacb66a173b849d357609bdae8e790524c077b82a7c6c72"
Jul 26 16:29:09.731359 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:09.731327    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-df352b-78957f86bd-2ls7x"]
Jul 26 16:29:09.731849 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:09.731612    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-df352b-78957f86bd-2ls7x" podUID="4cab2b78-d1c5-4598-b767-5e203100afa1" containerName="maas-api" containerID="cri-o://11181dc01e7f98715ccb8d5de85138bc841fe443b0fa574fb2d25227b639e49e" gracePeriod=30
Jul 26 16:29:09.752985 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:09.752953    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:29:09.778574 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:09.778541    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-df352b-676ffff8fd-q8kqn"]
Jul 26 16:29:09.778818 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:09.778790    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-df352b-676ffff8fd-q8kqn" podUID="eb8ba3f6-8d9b-4478-a127-a9802ef8ca7e" containerName="payload-processing" containerID="cri-o://485b512976b25ae240dac8cbe8c06e3ee78c0dee4ec0c2c2dbcfc776e19fe861" gracePeriod=30
Jul 26 16:29:09.788437 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:09.788397    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-df352b-88459dd-9t7bq"]
Jul 26 16:29:09.789142 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:09.788850    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-df352b-88459dd-9t7bq" podUID="d50bb5a9-4d7b-4053-b035-0e554ef18d0c" containerName="payload-pre-processing" containerID="cri-o://f9d189ef880c0ef9d24a9d8b4feba790f9f749f561e16a739706784ba8d7c057" gracePeriod=30
Jul 26 16:29:09.854315 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:09.854271    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:29:09.954940 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:09.954855    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:29:10.077489 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:10.077464    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-df352b-676ffff8fd-q8kqn"
Jul 26 16:29:10.081153 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:10.081134    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-df352b-78957f86bd-2ls7x"
Jul 26 16:29:10.101968 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:10.101946    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-df352b-88459dd-9t7bq"
Jul 26 16:29:10.128193 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:10.128166    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7xzv7\" (UniqueName: \"kubernetes.io/projected/4cab2b78-d1c5-4598-b767-5e203100afa1-kube-api-access-7xzv7\") pod \"4cab2b78-d1c5-4598-b767-5e203100afa1\" (UID: \"4cab2b78-d1c5-4598-b767-5e203100afa1\") "
Jul 26 16:29:10.128381 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:10.128210    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cab2b78-d1c5-4598-b767-5e203100afa1-openshift-service-ca\") pod \"4cab2b78-d1c5-4598-b767-5e203100afa1\" (UID: \"4cab2b78-d1c5-4598-b767-5e203100afa1\") "
Jul 26 16:29:10.128381 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:10.128242    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cab2b78-d1c5-4598-b767-5e203100afa1-maas-api-tls\") pod \"4cab2b78-d1c5-4598-b767-5e203100afa1\" (UID: \"4cab2b78-d1c5-4598-b767-5e203100afa1\") "
Jul 26 16:29:10.128381 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:10.128282    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d50bb5a9-4d7b-4053-b035-0e554ef18d0c-plugins-config-volume\") pod \"d50bb5a9-4d7b-4053-b035-0e554ef18d0c\" (UID: \"d50bb5a9-4d7b-4053-b035-0e554ef18d0c\") "
Jul 26 16:29:10.128381 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:10.128345    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t6wll\" (UniqueName: \"kubernetes.io/projected/d50bb5a9-4d7b-4053-b035-0e554ef18d0c-kube-api-access-t6wll\") pod \"d50bb5a9-4d7b-4053-b035-0e554ef18d0c\" (UID: \"d50bb5a9-4d7b-4053-b035-0e554ef18d0c\") "
Jul 26 16:29:10.128381 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:10.128375    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gch4p\" (UniqueName: \"kubernetes.io/projected/eb8ba3f6-8d9b-4478-a127-a9802ef8ca7e-kube-api-access-gch4p\") pod \"eb8ba3f6-8d9b-4478-a127-a9802ef8ca7e\" (UID: \"eb8ba3f6-8d9b-4478-a127-a9802ef8ca7e\") "
Jul 26 16:29:10.128661 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:10.128633    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb8ba3f6-8d9b-4478-a127-a9802ef8ca7e-plugins-config-volume\") pod \"eb8ba3f6-8d9b-4478-a127-a9802ef8ca7e\" (UID: \"eb8ba3f6-8d9b-4478-a127-a9802ef8ca7e\") "
Jul 26 16:29:10.128807 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:10.128782    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d50bb5a9-4d7b-4053-b035-0e554ef18d0c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d50bb5a9-4d7b-4053-b035-0e554ef18d0c" (UID: "d50bb5a9-4d7b-4053-b035-0e554ef18d0c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 26 16:29:10.129041 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:10.129013    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cab2b78-d1c5-4598-b767-5e203100afa1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4cab2b78-d1c5-4598-b767-5e203100afa1" (UID: "4cab2b78-d1c5-4598-b767-5e203100afa1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 26 16:29:10.129139 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:10.129044    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d50bb5a9-4d7b-4053-b035-0e554ef18d0c-plugins-config-volume\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:29:10.129244 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:10.129220    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb8ba3f6-8d9b-4478-a127-a9802ef8ca7e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "eb8ba3f6-8d9b-4478-a127-a9802ef8ca7e" (UID: "eb8ba3f6-8d9b-4478-a127-a9802ef8ca7e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 26 16:29:10.130522 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:10.130489    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb8ba3f6-8d9b-4478-a127-a9802ef8ca7e-kube-api-access-gch4p" (OuterVolumeSpecName: "kube-api-access-gch4p") pod "eb8ba3f6-8d9b-4478-a127-a9802ef8ca7e" (UID: "eb8ba3f6-8d9b-4478-a127-a9802ef8ca7e"). InnerVolumeSpecName "kube-api-access-gch4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 26 16:29:10.130621 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:10.130526    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cab2b78-d1c5-4598-b767-5e203100afa1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4cab2b78-d1c5-4598-b767-5e203100afa1" (UID: "4cab2b78-d1c5-4598-b767-5e203100afa1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 26 16:29:10.130621 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:10.130545    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d50bb5a9-4d7b-4053-b035-0e554ef18d0c-kube-api-access-t6wll" (OuterVolumeSpecName: "kube-api-access-t6wll") pod "d50bb5a9-4d7b-4053-b035-0e554ef18d0c" (UID: "d50bb5a9-4d7b-4053-b035-0e554ef18d0c"). InnerVolumeSpecName "kube-api-access-t6wll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 26 16:29:10.131038 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:10.131019    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cab2b78-d1c5-4598-b767-5e203100afa1-kube-api-access-7xzv7" (OuterVolumeSpecName: "kube-api-access-7xzv7") pod "4cab2b78-d1c5-4598-b767-5e203100afa1" (UID: "4cab2b78-d1c5-4598-b767-5e203100afa1"). InnerVolumeSpecName "kube-api-access-7xzv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 26 16:29:10.229708 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:10.229675    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6wll\" (UniqueName: \"kubernetes.io/projected/d50bb5a9-4d7b-4053-b035-0e554ef18d0c-kube-api-access-t6wll\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:29:10.229708 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:10.229702    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gch4p\" (UniqueName: \"kubernetes.io/projected/eb8ba3f6-8d9b-4478-a127-a9802ef8ca7e-kube-api-access-gch4p\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:29:10.229708 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:10.229711    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb8ba3f6-8d9b-4478-a127-a9802ef8ca7e-plugins-config-volume\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:29:10.229964 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:10.229721    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xzv7\" (UniqueName: \"kubernetes.io/projected/4cab2b78-d1c5-4598-b767-5e203100afa1-kube-api-access-7xzv7\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:29:10.229964 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:10.229730    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cab2b78-d1c5-4598-b767-5e203100afa1-openshift-service-ca\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:29:10.229964 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:10.229739    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cab2b78-d1c5-4598-b767-5e203100afa1-maas-api-tls\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:29:10.852552 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:10.852513    2563 generic.go:356] "Generic (PLEG): container finished" podID="d50bb5a9-4d7b-4053-b035-0e554ef18d0c" containerID="f9d189ef880c0ef9d24a9d8b4feba790f9f749f561e16a739706784ba8d7c057" exitCode=0
Jul 26 16:29:10.853066 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:10.852561    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-df352b-88459dd-9t7bq"
Jul 26 16:29:10.853066 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:10.852600    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-df352b-88459dd-9t7bq" event={"ID":"d50bb5a9-4d7b-4053-b035-0e554ef18d0c","Type":"ContainerDied","Data":"f9d189ef880c0ef9d24a9d8b4feba790f9f749f561e16a739706784ba8d7c057"}
Jul 26 16:29:10.853066 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:10.852660    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-df352b-88459dd-9t7bq" event={"ID":"d50bb5a9-4d7b-4053-b035-0e554ef18d0c","Type":"ContainerDied","Data":"8d52281872e8bc8c464c8569e25ac130b19001261da1b9912ba9b8740e110704"}
Jul 26 16:29:10.853066 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:10.852692    2563 scope.go:117] "RemoveContainer" containerID="f9d189ef880c0ef9d24a9d8b4feba790f9f749f561e16a739706784ba8d7c057"
Jul 26 16:29:10.854072 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:10.853885    2563 generic.go:356] "Generic (PLEG): container finished" podID="eb8ba3f6-8d9b-4478-a127-a9802ef8ca7e" containerID="485b512976b25ae240dac8cbe8c06e3ee78c0dee4ec0c2c2dbcfc776e19fe861" exitCode=0
Jul 26 16:29:10.854072 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:10.853915    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-df352b-676ffff8fd-q8kqn" event={"ID":"eb8ba3f6-8d9b-4478-a127-a9802ef8ca7e","Type":"ContainerDied","Data":"485b512976b25ae240dac8cbe8c06e3ee78c0dee4ec0c2c2dbcfc776e19fe861"}
Jul 26 16:29:10.854072 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:10.853936    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-df352b-676ffff8fd-q8kqn"
Jul 26 16:29:10.854072 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:10.853957    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-df352b-676ffff8fd-q8kqn" event={"ID":"eb8ba3f6-8d9b-4478-a127-a9802ef8ca7e","Type":"ContainerDied","Data":"dca84e57b861568775762be4c1d01e10211ec0bfc8aad419cfe8c1d01a2233b4"}
Jul 26 16:29:10.855632 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:10.855485    2563 generic.go:356] "Generic (PLEG): container finished" podID="4cab2b78-d1c5-4598-b767-5e203100afa1" containerID="11181dc01e7f98715ccb8d5de85138bc841fe443b0fa574fb2d25227b639e49e" exitCode=0
Jul 26 16:29:10.855632 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:10.855514    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-df352b-78957f86bd-2ls7x" event={"ID":"4cab2b78-d1c5-4598-b767-5e203100afa1","Type":"ContainerDied","Data":"11181dc01e7f98715ccb8d5de85138bc841fe443b0fa574fb2d25227b639e49e"}
Jul 26 16:29:10.855632 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:10.855527    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-df352b-78957f86bd-2ls7x"
Jul 26 16:29:10.855632 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:10.855550    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-df352b-78957f86bd-2ls7x" event={"ID":"4cab2b78-d1c5-4598-b767-5e203100afa1","Type":"ContainerDied","Data":"9a390b9aaa4546a254c16aba3e4d00b6b4fcc13dd6310b57b62dd94ec82c5ecb"}
Jul 26 16:29:10.862344 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:10.862330    2563 scope.go:117] "RemoveContainer" containerID="f9d189ef880c0ef9d24a9d8b4feba790f9f749f561e16a739706784ba8d7c057"
Jul 26 16:29:10.862588 ip-10-0-133-204 kubenswrapper[2563]: E0726 16:29:10.862572    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9d189ef880c0ef9d24a9d8b4feba790f9f749f561e16a739706784ba8d7c057\": container with ID starting with f9d189ef880c0ef9d24a9d8b4feba790f9f749f561e16a739706784ba8d7c057 not found: ID does not exist" containerID="f9d189ef880c0ef9d24a9d8b4feba790f9f749f561e16a739706784ba8d7c057"
Jul 26 16:29:10.862656 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:10.862594    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9d189ef880c0ef9d24a9d8b4feba790f9f749f561e16a739706784ba8d7c057"} err="failed to get container status \"f9d189ef880c0ef9d24a9d8b4feba790f9f749f561e16a739706784ba8d7c057\": rpc error: code = NotFound desc = could not find container \"f9d189ef880c0ef9d24a9d8b4feba790f9f749f561e16a739706784ba8d7c057\": container with ID starting with f9d189ef880c0ef9d24a9d8b4feba790f9f749f561e16a739706784ba8d7c057 not found: ID does not exist"
Jul 26 16:29:10.862656 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:10.862611    2563 scope.go:117] "RemoveContainer" containerID="485b512976b25ae240dac8cbe8c06e3ee78c0dee4ec0c2c2dbcfc776e19fe861"
Jul 26 16:29:10.870152 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:10.870137    2563 scope.go:117] "RemoveContainer" containerID="485b512976b25ae240dac8cbe8c06e3ee78c0dee4ec0c2c2dbcfc776e19fe861"
Jul 26 16:29:10.870398 ip-10-0-133-204 kubenswrapper[2563]: E0726 16:29:10.870376    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"485b512976b25ae240dac8cbe8c06e3ee78c0dee4ec0c2c2dbcfc776e19fe861\": container with ID starting with 485b512976b25ae240dac8cbe8c06e3ee78c0dee4ec0c2c2dbcfc776e19fe861 not found: ID does not exist" containerID="485b512976b25ae240dac8cbe8c06e3ee78c0dee4ec0c2c2dbcfc776e19fe861"
Jul 26 16:29:10.870460 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:10.870404    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"485b512976b25ae240dac8cbe8c06e3ee78c0dee4ec0c2c2dbcfc776e19fe861"} err="failed to get container status \"485b512976b25ae240dac8cbe8c06e3ee78c0dee4ec0c2c2dbcfc776e19fe861\": rpc error: code = NotFound desc = could not find container \"485b512976b25ae240dac8cbe8c06e3ee78c0dee4ec0c2c2dbcfc776e19fe861\": container with ID starting with 485b512976b25ae240dac8cbe8c06e3ee78c0dee4ec0c2c2dbcfc776e19fe861 not found: ID does not exist"
Jul 26 16:29:10.870460 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:10.870420    2563 scope.go:117] "RemoveContainer" containerID="11181dc01e7f98715ccb8d5de85138bc841fe443b0fa574fb2d25227b639e49e"
Jul 26 16:29:10.875595 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:10.875573    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-df352b-78957f86bd-2ls7x"]
Jul 26 16:29:10.878924 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:10.878904    2563 scope.go:117] "RemoveContainer" containerID="11181dc01e7f98715ccb8d5de85138bc841fe443b0fa574fb2d25227b639e49e"
Jul 26 16:29:10.879546 ip-10-0-133-204 kubenswrapper[2563]: E0726 16:29:10.879259    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11181dc01e7f98715ccb8d5de85138bc841fe443b0fa574fb2d25227b639e49e\": container with ID starting with 11181dc01e7f98715ccb8d5de85138bc841fe443b0fa574fb2d25227b639e49e not found: ID does not exist" containerID="11181dc01e7f98715ccb8d5de85138bc841fe443b0fa574fb2d25227b639e49e"
Jul 26 16:29:10.879546 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:10.879293    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11181dc01e7f98715ccb8d5de85138bc841fe443b0fa574fb2d25227b639e49e"} err="failed to get container status \"11181dc01e7f98715ccb8d5de85138bc841fe443b0fa574fb2d25227b639e49e\": rpc error: code = NotFound desc = could not find container \"11181dc01e7f98715ccb8d5de85138bc841fe443b0fa574fb2d25227b639e49e\": container with ID starting with 11181dc01e7f98715ccb8d5de85138bc841fe443b0fa574fb2d25227b639e49e not found: ID does not exist"
Jul 26 16:29:10.881754 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:10.881735    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-df352b-78957f86bd-2ls7x"]
Jul 26 16:29:10.902168 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:10.902140    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-df352b-676ffff8fd-q8kqn"]
Jul 26 16:29:10.906226 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:10.906204    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-df352b-676ffff8fd-q8kqn"]
Jul 26 16:29:10.916447 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:10.916426    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-df352b-88459dd-9t7bq"]
Jul 26 16:29:10.920402 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:10.920381    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-df352b-88459dd-9t7bq"]
Jul 26 16:29:12.373238 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:12.373207    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4cab2b78-d1c5-4598-b767-5e203100afa1" path="/var/lib/kubelet/pods/4cab2b78-d1c5-4598-b767-5e203100afa1/volumes"
Jul 26 16:29:12.373629 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:12.373567    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d50bb5a9-4d7b-4053-b035-0e554ef18d0c" path="/var/lib/kubelet/pods/d50bb5a9-4d7b-4053-b035-0e554ef18d0c/volumes"
Jul 26 16:29:12.373913 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:12.373899    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb8ba3f6-8d9b-4478-a127-a9802ef8ca7e" path="/var/lib/kubelet/pods/eb8ba3f6-8d9b-4478-a127-a9802ef8ca7e/volumes"
Jul 26 16:29:14.354237 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:14.354167    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-df352b-jtxrv"]
Jul 26 16:29:14.356520 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:14.356498    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-df352b-jtxrv"]
Jul 26 16:29:14.359372 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:14.359341    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:29:14.373204 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:14.373177    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec2cbe93-a640-4bdb-b942-b17273a3b89e" path="/var/lib/kubelet/pods/ec2cbe93-a640-4bdb-b942-b17273a3b89e/volumes"
Jul 26 16:29:14.756086 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:14.756052    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:29:20.525541 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:20.525505    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:29:20.602465 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:20.602430    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:29:20.808542 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:20.808431    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:29:21.037048 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:21.037016    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:29:21.405819 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:21.405787    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:29:21.743393 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:21.743352    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:29:22.007234 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:22.007155    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:29:25.159914 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:25.159879    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:29:25.244986 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:25.244949    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:29:25.247548 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:25.246637    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-76272d-5b6cb9986b-88n5f"]
Jul 26 16:29:25.247548 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:25.247218    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec2cbe93-a640-4bdb-b942-b17273a3b89e" containerName="revoke-keys"
Jul 26 16:29:25.247548 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:25.247238    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec2cbe93-a640-4bdb-b942-b17273a3b89e" containerName="revoke-keys"
Jul 26 16:29:25.247548 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:25.247259    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d50bb5a9-4d7b-4053-b035-0e554ef18d0c" containerName="payload-pre-processing"
Jul 26 16:29:25.247548 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:25.247268    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d50bb5a9-4d7b-4053-b035-0e554ef18d0c" containerName="payload-pre-processing"
Jul 26 16:29:25.247548 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:25.247288    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb8ba3f6-8d9b-4478-a127-a9802ef8ca7e" containerName="payload-processing"
Jul 26 16:29:25.247548 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:25.247296    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb8ba3f6-8d9b-4478-a127-a9802ef8ca7e" containerName="payload-processing"
Jul 26 16:29:25.247548 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:25.247307    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cab2b78-d1c5-4598-b767-5e203100afa1" containerName="maas-api"
Jul 26 16:29:25.247548 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:25.247314    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cab2b78-d1c5-4598-b767-5e203100afa1" containerName="maas-api"
Jul 26 16:29:25.247548 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:25.247393    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec2cbe93-a640-4bdb-b942-b17273a3b89e" containerName="revoke-keys"
Jul 26 16:29:25.247548 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:25.247403    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb8ba3f6-8d9b-4478-a127-a9802ef8ca7e" containerName="payload-processing"
Jul 26 16:29:25.247548 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:25.247412    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d50bb5a9-4d7b-4053-b035-0e554ef18d0c" containerName="payload-pre-processing"
Jul 26 16:29:25.247548 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:25.247420    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cab2b78-d1c5-4598-b767-5e203100afa1" containerName="maas-api"
Jul 26 16:29:25.250777 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:25.250744    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-76272d-5b6cb9986b-88n5f"
Jul 26 16:29:25.255463 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:25.255437    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-76272d-serving-cert\""
Jul 26 16:29:25.265818 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:25.265791    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-76272d-5b6cb9986b-88n5f"]
Jul 26 16:29:25.328175 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:25.328142    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:29:25.370302 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:25.370270    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq9tl\" (UniqueName: \"kubernetes.io/projected/28f0c6d3-1b43-4dca-a4d7-1a7e1d77f082-kube-api-access-nq9tl\") pod \"maas-api-e2e-shared-a-76272d-5b6cb9986b-88n5f\" (UID: \"28f0c6d3-1b43-4dca-a4d7-1a7e1d77f082\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-76272d-5b6cb9986b-88n5f"
Jul 26 16:29:25.370514 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:25.370325    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28f0c6d3-1b43-4dca-a4d7-1a7e1d77f082-openshift-service-ca\") pod \"maas-api-e2e-shared-a-76272d-5b6cb9986b-88n5f\" (UID: \"28f0c6d3-1b43-4dca-a4d7-1a7e1d77f082\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-76272d-5b6cb9986b-88n5f"
Jul 26 16:29:25.370514 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:25.370404    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28f0c6d3-1b43-4dca-a4d7-1a7e1d77f082-maas-api-tls\") pod \"maas-api-e2e-shared-a-76272d-5b6cb9986b-88n5f\" (UID: \"28f0c6d3-1b43-4dca-a4d7-1a7e1d77f082\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-76272d-5b6cb9986b-88n5f"
Jul 26 16:29:25.471832 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:25.471727    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq9tl\" (UniqueName: \"kubernetes.io/projected/28f0c6d3-1b43-4dca-a4d7-1a7e1d77f082-kube-api-access-nq9tl\") pod \"maas-api-e2e-shared-a-76272d-5b6cb9986b-88n5f\" (UID: \"28f0c6d3-1b43-4dca-a4d7-1a7e1d77f082\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-76272d-5b6cb9986b-88n5f"
Jul 26 16:29:25.471832 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:25.471813    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28f0c6d3-1b43-4dca-a4d7-1a7e1d77f082-openshift-service-ca\") pod \"maas-api-e2e-shared-a-76272d-5b6cb9986b-88n5f\" (UID: \"28f0c6d3-1b43-4dca-a4d7-1a7e1d77f082\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-76272d-5b6cb9986b-88n5f"
Jul 26 16:29:25.472061 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:25.471848    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28f0c6d3-1b43-4dca-a4d7-1a7e1d77f082-maas-api-tls\") pod \"maas-api-e2e-shared-a-76272d-5b6cb9986b-88n5f\" (UID: \"28f0c6d3-1b43-4dca-a4d7-1a7e1d77f082\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-76272d-5b6cb9986b-88n5f"
Jul 26 16:29:25.472061 ip-10-0-133-204 kubenswrapper[2563]: E0726 16:29:25.471949    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-76272d-serving-cert: secret "maas-api-e2e-shared-a-76272d-serving-cert" not found
Jul 26 16:29:25.472061 ip-10-0-133-204 kubenswrapper[2563]: E0726 16:29:25.472013    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28f0c6d3-1b43-4dca-a4d7-1a7e1d77f082-maas-api-tls podName:28f0c6d3-1b43-4dca-a4d7-1a7e1d77f082 nodeName:}" failed. No retries permitted until 2026-07-26 16:29:25.971998184 +0000 UTC m=+2358.076223970 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/28f0c6d3-1b43-4dca-a4d7-1a7e1d77f082-maas-api-tls") pod "maas-api-e2e-shared-a-76272d-5b6cb9986b-88n5f" (UID: "28f0c6d3-1b43-4dca-a4d7-1a7e1d77f082") : secret "maas-api-e2e-shared-a-76272d-serving-cert" not found
Jul 26 16:29:25.472435 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:25.472414    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28f0c6d3-1b43-4dca-a4d7-1a7e1d77f082-openshift-service-ca\") pod \"maas-api-e2e-shared-a-76272d-5b6cb9986b-88n5f\" (UID: \"28f0c6d3-1b43-4dca-a4d7-1a7e1d77f082\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-76272d-5b6cb9986b-88n5f"
Jul 26 16:29:25.486838 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:25.486802    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq9tl\" (UniqueName: \"kubernetes.io/projected/28f0c6d3-1b43-4dca-a4d7-1a7e1d77f082-kube-api-access-nq9tl\") pod \"maas-api-e2e-shared-a-76272d-5b6cb9986b-88n5f\" (UID: \"28f0c6d3-1b43-4dca-a4d7-1a7e1d77f082\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-76272d-5b6cb9986b-88n5f"
Jul 26 16:29:25.543997 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:25.543966    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:29:25.871155 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:25.871117    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:29:25.965683 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:25.965643    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:29:25.976701 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:25.976661    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28f0c6d3-1b43-4dca-a4d7-1a7e1d77f082-maas-api-tls\") pod \"maas-api-e2e-shared-a-76272d-5b6cb9986b-88n5f\" (UID: \"28f0c6d3-1b43-4dca-a4d7-1a7e1d77f082\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-76272d-5b6cb9986b-88n5f"
Jul 26 16:29:25.978405 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:25.978372    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-76272d-547fc4cdb8-wvgmm"]
Jul 26 16:29:25.979287 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:25.979265    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28f0c6d3-1b43-4dca-a4d7-1a7e1d77f082-maas-api-tls\") pod \"maas-api-e2e-shared-a-76272d-5b6cb9986b-88n5f\" (UID: \"28f0c6d3-1b43-4dca-a4d7-1a7e1d77f082\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-76272d-5b6cb9986b-88n5f"
Jul 26 16:29:25.981836 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:25.981820    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-76272d-547fc4cdb8-wvgmm"
Jul 26 16:29:25.983946 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:25.983926    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-76272d\""
Jul 26 16:29:25.984072 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:25.984024    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-76272d-dockercfg-pztl6\""
Jul 26 16:29:25.996166 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:25.996141    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-76272d-547fc4cdb8-wvgmm"]
Jul 26 16:29:26.008431 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:26.008401    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-76272d-785765d686-xmmq6"]
Jul 26 16:29:26.012131 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:26.012103    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-76272d-785765d686-xmmq6"
Jul 26 16:29:26.022509 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:26.022479    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-76272d-785765d686-xmmq6"]
Jul 26 16:29:26.078010 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:26.077964    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfx5r\" (UniqueName: \"kubernetes.io/projected/60187994-88fd-4943-9cb8-5316c18157de-kube-api-access-dfx5r\") pod \"payload-processing-e2e-shared-a-76272d-785765d686-xmmq6\" (UID: \"60187994-88fd-4943-9cb8-5316c18157de\") " pod="openshift-ingress/payload-processing-e2e-shared-a-76272d-785765d686-xmmq6"
Jul 26 16:29:26.078185 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:26.078036    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd05518b-df0b-4078-9c51-1c74eec82cfa-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-76272d-547fc4cdb8-wvgmm\" (UID: \"fd05518b-df0b-4078-9c51-1c74eec82cfa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-76272d-547fc4cdb8-wvgmm"
Jul 26 16:29:26.078185 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:26.078089    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ks84\" (UniqueName: \"kubernetes.io/projected/fd05518b-df0b-4078-9c51-1c74eec82cfa-kube-api-access-7ks84\") pod \"payload-pre-processing-e2e-shared-a-76272d-547fc4cdb8-wvgmm\" (UID: \"fd05518b-df0b-4078-9c51-1c74eec82cfa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-76272d-547fc4cdb8-wvgmm"
Jul 26 16:29:26.078319 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:26.078175    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60187994-88fd-4943-9cb8-5316c18157de-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-76272d-785765d686-xmmq6\" (UID: \"60187994-88fd-4943-9cb8-5316c18157de\") " pod="openshift-ingress/payload-processing-e2e-shared-a-76272d-785765d686-xmmq6"
Jul 26 16:29:26.162332 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:26.162242    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-76272d-5b6cb9986b-88n5f"
Jul 26 16:29:26.178810 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:26.178755    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfx5r\" (UniqueName: \"kubernetes.io/projected/60187994-88fd-4943-9cb8-5316c18157de-kube-api-access-dfx5r\") pod \"payload-processing-e2e-shared-a-76272d-785765d686-xmmq6\" (UID: \"60187994-88fd-4943-9cb8-5316c18157de\") " pod="openshift-ingress/payload-processing-e2e-shared-a-76272d-785765d686-xmmq6"
Jul 26 16:29:26.179017 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:26.178852    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd05518b-df0b-4078-9c51-1c74eec82cfa-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-76272d-547fc4cdb8-wvgmm\" (UID: \"fd05518b-df0b-4078-9c51-1c74eec82cfa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-76272d-547fc4cdb8-wvgmm"
Jul 26 16:29:26.179017 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:26.178912    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ks84\" (UniqueName: \"kubernetes.io/projected/fd05518b-df0b-4078-9c51-1c74eec82cfa-kube-api-access-7ks84\") pod \"payload-pre-processing-e2e-shared-a-76272d-547fc4cdb8-wvgmm\" (UID: \"fd05518b-df0b-4078-9c51-1c74eec82cfa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-76272d-547fc4cdb8-wvgmm"
Jul 26 16:29:26.179017 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:26.178967    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60187994-88fd-4943-9cb8-5316c18157de-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-76272d-785765d686-xmmq6\" (UID: \"60187994-88fd-4943-9cb8-5316c18157de\") " pod="openshift-ingress/payload-processing-e2e-shared-a-76272d-785765d686-xmmq6"
Jul 26 16:29:26.179682 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:26.179662    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd05518b-df0b-4078-9c51-1c74eec82cfa-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-76272d-547fc4cdb8-wvgmm\" (UID: \"fd05518b-df0b-4078-9c51-1c74eec82cfa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-76272d-547fc4cdb8-wvgmm"
Jul 26 16:29:26.179751 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:26.179663    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60187994-88fd-4943-9cb8-5316c18157de-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-76272d-785765d686-xmmq6\" (UID: \"60187994-88fd-4943-9cb8-5316c18157de\") " pod="openshift-ingress/payload-processing-e2e-shared-a-76272d-785765d686-xmmq6"
Jul 26 16:29:26.188135 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:26.188100    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfx5r\" (UniqueName: \"kubernetes.io/projected/60187994-88fd-4943-9cb8-5316c18157de-kube-api-access-dfx5r\") pod \"payload-processing-e2e-shared-a-76272d-785765d686-xmmq6\" (UID: \"60187994-88fd-4943-9cb8-5316c18157de\") " pod="openshift-ingress/payload-processing-e2e-shared-a-76272d-785765d686-xmmq6"
Jul 26 16:29:26.190155 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:26.190125    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ks84\" (UniqueName: \"kubernetes.io/projected/fd05518b-df0b-4078-9c51-1c74eec82cfa-kube-api-access-7ks84\") pod \"payload-pre-processing-e2e-shared-a-76272d-547fc4cdb8-wvgmm\" (UID: \"fd05518b-df0b-4078-9c51-1c74eec82cfa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-76272d-547fc4cdb8-wvgmm"
Jul 26 16:29:26.300007 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:26.299975    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-76272d-547fc4cdb8-wvgmm"
Jul 26 16:29:26.314170 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:26.314144    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-76272d-5b6cb9986b-88n5f"]
Jul 26 16:29:26.316704 ip-10-0-133-204 kubenswrapper[2563]: W0726 16:29:26.316673    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28f0c6d3_1b43_4dca_a4d7_1a7e1d77f082.slice/crio-23e5e779411046d1f4eb9ea44ea50e2245649585fedf0befa61c1daa3a703635 WatchSource:0}: Error finding container 23e5e779411046d1f4eb9ea44ea50e2245649585fedf0befa61c1daa3a703635: Status 404 returned error can't find the container with id 23e5e779411046d1f4eb9ea44ea50e2245649585fedf0befa61c1daa3a703635
Jul 26 16:29:26.323794 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:26.323751    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-76272d-785765d686-xmmq6"
Jul 26 16:29:26.460002 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:26.459966    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-76272d-547fc4cdb8-wvgmm"]
Jul 26 16:29:26.463603 ip-10-0-133-204 kubenswrapper[2563]: W0726 16:29:26.463574    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd05518b_df0b_4078_9c51_1c74eec82cfa.slice/crio-16d32ed704ac42d575c43ce148eabd046de6534cdb3c88ad5861bcc00725dc95 WatchSource:0}: Error finding container 16d32ed704ac42d575c43ce148eabd046de6534cdb3c88ad5861bcc00725dc95: Status 404 returned error can't find the container with id 16d32ed704ac42d575c43ce148eabd046de6534cdb3c88ad5861bcc00725dc95
Jul 26 16:29:26.488138 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:26.487875    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-76272d-785765d686-xmmq6"]
Jul 26 16:29:26.493284 ip-10-0-133-204 kubenswrapper[2563]: W0726 16:29:26.493253    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60187994_88fd_4943_9cb8_5316c18157de.slice/crio-eb80020814df5a11691204045f78bd04ee101b3477751a848876b9203c28828f WatchSource:0}: Error finding container eb80020814df5a11691204045f78bd04ee101b3477751a848876b9203c28828f: Status 404 returned error can't find the container with id eb80020814df5a11691204045f78bd04ee101b3477751a848876b9203c28828f
Jul 26 16:29:26.938239 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:26.938139    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-76272d-5b6cb9986b-88n5f" event={"ID":"28f0c6d3-1b43-4dca-a4d7-1a7e1d77f082","Type":"ContainerStarted","Data":"1374c914dd9f29e8adfb9bbed79f8165f9b82f9029f0304a83f949e3b59d7062"}
Jul 26 16:29:26.938239 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:26.938207    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-76272d-5b6cb9986b-88n5f"
Jul 26 16:29:26.938239 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:26.938235    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-76272d-5b6cb9986b-88n5f" event={"ID":"28f0c6d3-1b43-4dca-a4d7-1a7e1d77f082","Type":"ContainerStarted","Data":"23e5e779411046d1f4eb9ea44ea50e2245649585fedf0befa61c1daa3a703635"}
Jul 26 16:29:26.939648 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:26.939623    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-76272d-785765d686-xmmq6" event={"ID":"60187994-88fd-4943-9cb8-5316c18157de","Type":"ContainerStarted","Data":"0fda0355ee14af9ac8ca6d7aa3562e54c2f1b757b2afe48cc56babe35ffb72bf"}
Jul 26 16:29:26.939784 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:26.939663    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-76272d-785765d686-xmmq6" event={"ID":"60187994-88fd-4943-9cb8-5316c18157de","Type":"ContainerStarted","Data":"eb80020814df5a11691204045f78bd04ee101b3477751a848876b9203c28828f"}
Jul 26 16:29:26.939784 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:26.939694    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-76272d-785765d686-xmmq6"
Jul 26 16:29:26.941042 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:26.941018    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-76272d-547fc4cdb8-wvgmm" event={"ID":"fd05518b-df0b-4078-9c51-1c74eec82cfa","Type":"ContainerStarted","Data":"62179c84212d79db85bd45f4bbddd62fd2e277915104e796445fdf97cffc3c1e"}
Jul 26 16:29:26.941134 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:26.941057    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-76272d-547fc4cdb8-wvgmm" event={"ID":"fd05518b-df0b-4078-9c51-1c74eec82cfa","Type":"ContainerStarted","Data":"16d32ed704ac42d575c43ce148eabd046de6534cdb3c88ad5861bcc00725dc95"}
Jul 26 16:29:26.941134 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:26.941125    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-76272d-547fc4cdb8-wvgmm"
Jul 26 16:29:26.954261 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:26.954208    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-76272d-5b6cb9986b-88n5f" podStartSLOduration=1.655441537 podStartE2EDuration="1.954191213s" totalImagesPullingTime="298.749676ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:29:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:29:26.954002485 +0000 UTC m=+2359.058228303" watchObservedRunningTime="2026-07-26 16:29:26.954191213 +0000 UTC m=+2359.058417024"
Jul 26 16:29:26.970322 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:26.970269    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-76272d-785765d686-xmmq6" podStartSLOduration=1.970253342 podStartE2EDuration="1.970253342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:29:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:29:26.968992228 +0000 UTC m=+2359.073218037" watchObservedRunningTime="2026-07-26 16:29:26.970253342 +0000 UTC m=+2359.074479129"
Jul 26 16:29:26.986651 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:26.986599    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-76272d-547fc4cdb8-wvgmm" podStartSLOduration=1.986583172 podStartE2EDuration="1.986583172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:29:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:29:26.983812694 +0000 UTC m=+2359.088038502" watchObservedRunningTime="2026-07-26 16:29:26.986583172 +0000 UTC m=+2359.090808981"
Jul 26 16:29:29.724046 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:29.724010    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:29:30.084307 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:30.084211    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:29:30.432841 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:30.432683    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:29:30.726451 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:30.726405    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:29:32.952360 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:32.952332    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-76272d-5b6cb9986b-88n5f"
Jul 26 16:29:33.309299 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:33.309261    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:29:33.405556 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:33.405522    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-99124e-587f6b5499-778wz"]
Jul 26 16:29:33.421138 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:33.421110    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-99124e-587f6b5499-778wz"
Jul 26 16:29:33.421809 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:33.421172    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-99124e-587f6b5499-778wz"]
Jul 26 16:29:33.424576 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:33.424556    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-99124e-serving-cert\""
Jul 26 16:29:33.426786 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:33.426716    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:29:33.516184 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:33.516148    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:29:33.550100 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:33.550063    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed4e99e6-92b1-49cb-a4f0-a4f354b37462-maas-api-tls\") pod \"maas-api-e2e-shared-b-99124e-587f6b5499-778wz\" (UID: \"ed4e99e6-92b1-49cb-a4f0-a4f354b37462\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-99124e-587f6b5499-778wz"
Jul 26 16:29:33.550100 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:33.550102    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlbbl\" (UniqueName: \"kubernetes.io/projected/ed4e99e6-92b1-49cb-a4f0-a4f354b37462-kube-api-access-rlbbl\") pod \"maas-api-e2e-shared-b-99124e-587f6b5499-778wz\" (UID: \"ed4e99e6-92b1-49cb-a4f0-a4f354b37462\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-99124e-587f6b5499-778wz"
Jul 26 16:29:33.550313 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:33.550128    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed4e99e6-92b1-49cb-a4f0-a4f354b37462-openshift-service-ca\") pod \"maas-api-e2e-shared-b-99124e-587f6b5499-778wz\" (UID: \"ed4e99e6-92b1-49cb-a4f0-a4f354b37462\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-99124e-587f6b5499-778wz"
Jul 26 16:29:33.651142 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:33.651052    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed4e99e6-92b1-49cb-a4f0-a4f354b37462-maas-api-tls\") pod \"maas-api-e2e-shared-b-99124e-587f6b5499-778wz\" (UID: \"ed4e99e6-92b1-49cb-a4f0-a4f354b37462\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-99124e-587f6b5499-778wz"
Jul 26 16:29:33.651142 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:33.651084    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlbbl\" (UniqueName: \"kubernetes.io/projected/ed4e99e6-92b1-49cb-a4f0-a4f354b37462-kube-api-access-rlbbl\") pod \"maas-api-e2e-shared-b-99124e-587f6b5499-778wz\" (UID: \"ed4e99e6-92b1-49cb-a4f0-a4f354b37462\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-99124e-587f6b5499-778wz"
Jul 26 16:29:33.651142 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:33.651112    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed4e99e6-92b1-49cb-a4f0-a4f354b37462-openshift-service-ca\") pod \"maas-api-e2e-shared-b-99124e-587f6b5499-778wz\" (UID: \"ed4e99e6-92b1-49cb-a4f0-a4f354b37462\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-99124e-587f6b5499-778wz"
Jul 26 16:29:33.651437 ip-10-0-133-204 kubenswrapper[2563]: E0726 16:29:33.651231    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-99124e-serving-cert: secret "maas-api-e2e-shared-b-99124e-serving-cert" not found
Jul 26 16:29:33.651437 ip-10-0-133-204 kubenswrapper[2563]: E0726 16:29:33.651303    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed4e99e6-92b1-49cb-a4f0-a4f354b37462-maas-api-tls podName:ed4e99e6-92b1-49cb-a4f0-a4f354b37462 nodeName:}" failed. No retries permitted until 2026-07-26 16:29:34.151283472 +0000 UTC m=+2366.255509264 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ed4e99e6-92b1-49cb-a4f0-a4f354b37462-maas-api-tls") pod "maas-api-e2e-shared-b-99124e-587f6b5499-778wz" (UID: "ed4e99e6-92b1-49cb-a4f0-a4f354b37462") : secret "maas-api-e2e-shared-b-99124e-serving-cert" not found
Jul 26 16:29:33.651755 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:33.651736    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed4e99e6-92b1-49cb-a4f0-a4f354b37462-openshift-service-ca\") pod \"maas-api-e2e-shared-b-99124e-587f6b5499-778wz\" (UID: \"ed4e99e6-92b1-49cb-a4f0-a4f354b37462\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-99124e-587f6b5499-778wz"
Jul 26 16:29:33.660517 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:33.660485    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlbbl\" (UniqueName: \"kubernetes.io/projected/ed4e99e6-92b1-49cb-a4f0-a4f354b37462-kube-api-access-rlbbl\") pod \"maas-api-e2e-shared-b-99124e-587f6b5499-778wz\" (UID: \"ed4e99e6-92b1-49cb-a4f0-a4f354b37462\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-99124e-587f6b5499-778wz"
Jul 26 16:29:33.869998 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:33.869962    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:29:34.063070 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:34.063041    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:29:34.156534 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:34.156498    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed4e99e6-92b1-49cb-a4f0-a4f354b37462-maas-api-tls\") pod \"maas-api-e2e-shared-b-99124e-587f6b5499-778wz\" (UID: \"ed4e99e6-92b1-49cb-a4f0-a4f354b37462\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-99124e-587f6b5499-778wz"
Jul 26 16:29:34.158955 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:34.158935    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed4e99e6-92b1-49cb-a4f0-a4f354b37462-maas-api-tls\") pod \"maas-api-e2e-shared-b-99124e-587f6b5499-778wz\" (UID: \"ed4e99e6-92b1-49cb-a4f0-a4f354b37462\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-99124e-587f6b5499-778wz"
Jul 26 16:29:34.188681 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:34.188649    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:29:34.288954 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:34.288914    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-99124e-76bff4c976-59pcc"]
Jul 26 16:29:34.292731 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:34.292710    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-99124e-76bff4c976-59pcc"
Jul 26 16:29:34.298899 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:34.298876    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-99124e-dockercfg-gfcgw\""
Jul 26 16:29:34.299026 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:34.298876    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-99124e\""
Jul 26 16:29:34.307586 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:34.307563    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-99124e-76bff4c976-59pcc"]
Jul 26 16:29:34.335960 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:34.335874    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-99124e-587f6b5499-778wz"
Jul 26 16:29:34.373649 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:34.373623    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-99124e-59545f7886-6874r"]
Jul 26 16:29:34.378793 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:34.378745    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-99124e-59545f7886-6874r"
Jul 26 16:29:34.387245 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:34.387209    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-99124e-59545f7886-6874r"]
Jul 26 16:29:34.459513 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:34.459463    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0727f694-727c-400d-98ba-e4309db6dd2e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-99124e-59545f7886-6874r\" (UID: \"0727f694-727c-400d-98ba-e4309db6dd2e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-99124e-59545f7886-6874r"
Jul 26 16:29:34.459692 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:34.459516    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqz2c\" (UniqueName: \"kubernetes.io/projected/0b75076d-7307-4c59-8c10-bc84eda0450b-kube-api-access-qqz2c\") pod \"payload-pre-processing-e2e-shared-b-99124e-76bff4c976-59pcc\" (UID: \"0b75076d-7307-4c59-8c10-bc84eda0450b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-99124e-76bff4c976-59pcc"
Jul 26 16:29:34.459692 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:34.459611    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4vn8\" (UniqueName: \"kubernetes.io/projected/0727f694-727c-400d-98ba-e4309db6dd2e-kube-api-access-c4vn8\") pod \"payload-processing-e2e-shared-b-99124e-59545f7886-6874r\" (UID: \"0727f694-727c-400d-98ba-e4309db6dd2e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-99124e-59545f7886-6874r"
Jul 26 16:29:34.459692 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:34.459658    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b75076d-7307-4c59-8c10-bc84eda0450b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-99124e-76bff4c976-59pcc\" (UID: \"0b75076d-7307-4c59-8c10-bc84eda0450b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-99124e-76bff4c976-59pcc"
Jul 26 16:29:34.495060 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:34.495032    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-99124e-587f6b5499-778wz"]
Jul 26 16:29:34.497179 ip-10-0-133-204 kubenswrapper[2563]: W0726 16:29:34.497155    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded4e99e6_92b1_49cb_a4f0_a4f354b37462.slice/crio-c54392f7b86e6fa9b2542ba98a499a8676b7c1aaafa4e468a7166ec171d8be64 WatchSource:0}: Error finding container c54392f7b86e6fa9b2542ba98a499a8676b7c1aaafa4e468a7166ec171d8be64: Status 404 returned error can't find the container with id c54392f7b86e6fa9b2542ba98a499a8676b7c1aaafa4e468a7166ec171d8be64
Jul 26 16:29:34.560849 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:34.560794    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0727f694-727c-400d-98ba-e4309db6dd2e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-99124e-59545f7886-6874r\" (UID: \"0727f694-727c-400d-98ba-e4309db6dd2e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-99124e-59545f7886-6874r"
Jul 26 16:29:34.561032 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:34.560862    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqz2c\" (UniqueName: \"kubernetes.io/projected/0b75076d-7307-4c59-8c10-bc84eda0450b-kube-api-access-qqz2c\") pod \"payload-pre-processing-e2e-shared-b-99124e-76bff4c976-59pcc\" (UID: \"0b75076d-7307-4c59-8c10-bc84eda0450b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-99124e-76bff4c976-59pcc"
Jul 26 16:29:34.561032 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:34.560934    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4vn8\" (UniqueName: \"kubernetes.io/projected/0727f694-727c-400d-98ba-e4309db6dd2e-kube-api-access-c4vn8\") pod \"payload-processing-e2e-shared-b-99124e-59545f7886-6874r\" (UID: \"0727f694-727c-400d-98ba-e4309db6dd2e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-99124e-59545f7886-6874r"
Jul 26 16:29:34.561032 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:34.560985    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b75076d-7307-4c59-8c10-bc84eda0450b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-99124e-76bff4c976-59pcc\" (UID: \"0b75076d-7307-4c59-8c10-bc84eda0450b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-99124e-76bff4c976-59pcc"
Jul 26 16:29:34.561482 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:34.561462    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0727f694-727c-400d-98ba-e4309db6dd2e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-99124e-59545f7886-6874r\" (UID: \"0727f694-727c-400d-98ba-e4309db6dd2e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-99124e-59545f7886-6874r"
Jul 26 16:29:34.561713 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:34.561691    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b75076d-7307-4c59-8c10-bc84eda0450b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-99124e-76bff4c976-59pcc\" (UID: \"0b75076d-7307-4c59-8c10-bc84eda0450b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-99124e-76bff4c976-59pcc"
Jul 26 16:29:34.570160 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:34.570129    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqz2c\" (UniqueName: \"kubernetes.io/projected/0b75076d-7307-4c59-8c10-bc84eda0450b-kube-api-access-qqz2c\") pod \"payload-pre-processing-e2e-shared-b-99124e-76bff4c976-59pcc\" (UID: \"0b75076d-7307-4c59-8c10-bc84eda0450b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-99124e-76bff4c976-59pcc"
Jul 26 16:29:34.571186 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:34.571163    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4vn8\" (UniqueName: \"kubernetes.io/projected/0727f694-727c-400d-98ba-e4309db6dd2e-kube-api-access-c4vn8\") pod \"payload-processing-e2e-shared-b-99124e-59545f7886-6874r\" (UID: \"0727f694-727c-400d-98ba-e4309db6dd2e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-99124e-59545f7886-6874r"
Jul 26 16:29:34.604221 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:34.604144    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-99124e-76bff4c976-59pcc"
Jul 26 16:29:34.698093 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:34.698062    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-99124e-59545f7886-6874r"
Jul 26 16:29:34.754806 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:34.754644    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-99124e-76bff4c976-59pcc"]
Jul 26 16:29:34.871085 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:34.871008    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-99124e-59545f7886-6874r"]
Jul 26 16:29:34.873825 ip-10-0-133-204 kubenswrapper[2563]: W0726 16:29:34.873799    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0727f694_727c_400d_98ba_e4309db6dd2e.slice/crio-88b0a8810d1921e46d05833ddbc07ae4d441efb8215e6b2fd73e5002f18409af WatchSource:0}: Error finding container 88b0a8810d1921e46d05833ddbc07ae4d441efb8215e6b2fd73e5002f18409af: Status 404 returned error can't find the container with id 88b0a8810d1921e46d05833ddbc07ae4d441efb8215e6b2fd73e5002f18409af
Jul 26 16:29:34.983911 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:34.983882    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-99124e-587f6b5499-778wz" event={"ID":"ed4e99e6-92b1-49cb-a4f0-a4f354b37462","Type":"ContainerStarted","Data":"5558aab8a2d04c1473a4b64d05b4d2b7dc6a37d26efd4fb27b11b29d591bea5b"}
Jul 26 16:29:34.984068 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:34.983930    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-99124e-587f6b5499-778wz" event={"ID":"ed4e99e6-92b1-49cb-a4f0-a4f354b37462","Type":"ContainerStarted","Data":"c54392f7b86e6fa9b2542ba98a499a8676b7c1aaafa4e468a7166ec171d8be64"}
Jul 26 16:29:34.984068 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:34.983983    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-99124e-587f6b5499-778wz"
Jul 26 16:29:34.989122 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:34.987730    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-99124e-76bff4c976-59pcc" event={"ID":"0b75076d-7307-4c59-8c10-bc84eda0450b","Type":"ContainerStarted","Data":"8c6eed874fb6f3201a74c27435a6bf68496926b7faf1967171ca7d9affd56753"}
Jul 26 16:29:34.989122 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:34.987797    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-99124e-76bff4c976-59pcc" event={"ID":"0b75076d-7307-4c59-8c10-bc84eda0450b","Type":"ContainerStarted","Data":"d6bd95bfdf90b4673fa482e533ddd79f272226cd52ae3080e2a2113b8933615f"}
Jul 26 16:29:34.989122 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:34.988276    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-99124e-76bff4c976-59pcc"
Jul 26 16:29:34.990951 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:34.990349    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-99124e-59545f7886-6874r" event={"ID":"0727f694-727c-400d-98ba-e4309db6dd2e","Type":"ContainerStarted","Data":"eabdc8185e9d6af7a934a315cebf170a441ed087850789a420022117f400217f"}
Jul 26 16:29:34.990951 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:34.990392    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-99124e-59545f7886-6874r" event={"ID":"0727f694-727c-400d-98ba-e4309db6dd2e","Type":"ContainerStarted","Data":"88b0a8810d1921e46d05833ddbc07ae4d441efb8215e6b2fd73e5002f18409af"}
Jul 26 16:29:34.990951 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:34.990900    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-99124e-59545f7886-6874r"
Jul 26 16:29:35.001883 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:35.001827    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-99124e-587f6b5499-778wz" podStartSLOduration=1.6104372420000002 podStartE2EDuration="2.001809601s" totalImagesPullingTime="391.372359ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:29:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:29:35.000966343 +0000 UTC m=+2367.105192155" watchObservedRunningTime="2026-07-26 16:29:35.001809601 +0000 UTC m=+2367.106035410"
Jul 26 16:29:35.017617 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:35.017563    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-99124e-76bff4c976-59pcc" podStartSLOduration=1.017546568 podStartE2EDuration="1.017546568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:29:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:29:35.01700726 +0000 UTC m=+2367.121233070" watchObservedRunningTime="2026-07-26 16:29:35.017546568 +0000 UTC m=+2367.121772377"
Jul 26 16:29:35.036136 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:35.036080    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-99124e-59545f7886-6874r" podStartSLOduration=1.036059836 podStartE2EDuration="1.036059836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:29:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:29:35.034853874 +0000 UTC m=+2367.139079688" watchObservedRunningTime="2026-07-26 16:29:35.036059836 +0000 UTC m=+2367.140285647"
Jul 26 16:29:37.740221 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:37.740189    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:29:37.949131 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:37.949099    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-76272d-785765d686-xmmq6"
Jul 26 16:29:37.949301 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:37.949277    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-76272d-547fc4cdb8-wvgmm"
Jul 26 16:29:41.002517 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:41.002487    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-99124e-587f6b5499-778wz"
Jul 26 16:29:47.005079 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:47.004985    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-99124e-76bff4c976-59pcc" podUID="0b75076d-7307-4c59-8c10-bc84eda0450b" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.75:9004: connect: connection refused"
Jul 26 16:29:47.005487 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:47.005465    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-99124e-59545f7886-6874r"
Jul 26 16:29:50.785968 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:50.785929    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:29:57.004840 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:29:57.004809    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-99124e-76bff4c976-59pcc"
Jul 26 16:30:24.186057 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:30:24.186008    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:30:31.599720 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:30:31.599588    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:30:55.044606 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:30:55.044542    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:31:09.909215 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:31:09.909175    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:31:17.106217 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:31:17.106161    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:31:17.180998 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:31:17.180960    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:32:06.995317 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:32:06.995231    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:33:49.205318 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:33:49.205235    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:35:08.716969 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:35:08.716888    2563 scope.go:117] "RemoveContainer" containerID="3aeabdb0012b605de8815d5b40b53b4b74989fdf33935fe1427a557b0a9eb601"
Jul 26 16:35:12.380069 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:35:12.380033    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:35:13.398905 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:35:13.398861    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:35:20.240604 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:35:20.240572    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:35:40.101994 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:35:40.101955    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:35:53.407334 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:35:53.407285    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:36:27.297043 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:36:27.296985    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:37:03.915376 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:37:03.915255    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:37:04.851306 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:37:04.851275    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:37:07.137739 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:37:07.137684    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:37:09.101128 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:37:09.101091    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:37:34.094716 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:37:34.094678    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:37:46.355572 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:37:46.355536    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:38:42.518859 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:38:42.518823    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:38:43.529894 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:38:43.529859    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:39:03.301149 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:03.301097    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:39:04.945624 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:04.945593    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:39:07.097450 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:07.097410    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:39:07.239130 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:07.239100    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:39:09.503595 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:09.503558    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bdfbca-5d9d85bd8b-7cl5p"]
Jul 26 16:39:09.507291 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:09.507271    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bdfbca-5d9d85bd8b-7cl5p"
Jul 26 16:39:09.509988 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:09.509970    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-bdfbca-serving-cert\""
Jul 26 16:39:09.524239 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:09.524213    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fefb5476-d140-46e0-87cf-128f5d761535-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-bdfbca-5d9d85bd8b-7cl5p\" (UID: \"fefb5476-d140-46e0-87cf-128f5d761535\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bdfbca-5d9d85bd8b-7cl5p"
Jul 26 16:39:09.524348 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:09.524253    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tgv5\" (UniqueName: \"kubernetes.io/projected/fefb5476-d140-46e0-87cf-128f5d761535-kube-api-access-4tgv5\") pod \"maas-api-e2e-ipp-a-bdfbca-5d9d85bd8b-7cl5p\" (UID: \"fefb5476-d140-46e0-87cf-128f5d761535\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bdfbca-5d9d85bd8b-7cl5p"
Jul 26 16:39:09.524348 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:09.524270    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fefb5476-d140-46e0-87cf-128f5d761535-maas-api-tls\") pod \"maas-api-e2e-ipp-a-bdfbca-5d9d85bd8b-7cl5p\" (UID: \"fefb5476-d140-46e0-87cf-128f5d761535\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bdfbca-5d9d85bd8b-7cl5p"
Jul 26 16:39:09.533706 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:09.533678    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:39:09.535512 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:09.535485    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bdfbca-5d9d85bd8b-7cl5p"]
Jul 26 16:39:09.624901 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:09.624867    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fefb5476-d140-46e0-87cf-128f5d761535-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-bdfbca-5d9d85bd8b-7cl5p\" (UID: \"fefb5476-d140-46e0-87cf-128f5d761535\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bdfbca-5d9d85bd8b-7cl5p"
Jul 26 16:39:09.625084 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:09.624913    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tgv5\" (UniqueName: \"kubernetes.io/projected/fefb5476-d140-46e0-87cf-128f5d761535-kube-api-access-4tgv5\") pod \"maas-api-e2e-ipp-a-bdfbca-5d9d85bd8b-7cl5p\" (UID: \"fefb5476-d140-46e0-87cf-128f5d761535\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bdfbca-5d9d85bd8b-7cl5p"
Jul 26 16:39:09.625084 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:09.625029    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fefb5476-d140-46e0-87cf-128f5d761535-maas-api-tls\") pod \"maas-api-e2e-ipp-a-bdfbca-5d9d85bd8b-7cl5p\" (UID: \"fefb5476-d140-46e0-87cf-128f5d761535\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bdfbca-5d9d85bd8b-7cl5p"
Jul 26 16:39:09.625494 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:09.625475    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fefb5476-d140-46e0-87cf-128f5d761535-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-bdfbca-5d9d85bd8b-7cl5p\" (UID: \"fefb5476-d140-46e0-87cf-128f5d761535\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bdfbca-5d9d85bd8b-7cl5p"
Jul 26 16:39:09.627378 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:09.627355    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fefb5476-d140-46e0-87cf-128f5d761535-maas-api-tls\") pod \"maas-api-e2e-ipp-a-bdfbca-5d9d85bd8b-7cl5p\" (UID: \"fefb5476-d140-46e0-87cf-128f5d761535\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bdfbca-5d9d85bd8b-7cl5p"
Jul 26 16:39:09.642634 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:09.642615    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tgv5\" (UniqueName: \"kubernetes.io/projected/fefb5476-d140-46e0-87cf-128f5d761535-kube-api-access-4tgv5\") pod \"maas-api-e2e-ipp-a-bdfbca-5d9d85bd8b-7cl5p\" (UID: \"fefb5476-d140-46e0-87cf-128f5d761535\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bdfbca-5d9d85bd8b-7cl5p"
Jul 26 16:39:09.678168 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:09.678125    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:39:09.817606 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:09.817522    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bdfbca-5d9d85bd8b-7cl5p"
Jul 26 16:39:09.885836 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:09.885709    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:39:09.966610 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:09.966570    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bdfbca-5d9d85bd8b-7cl5p"]
Jul 26 16:39:09.969828 ip-10-0-133-204 kubenswrapper[2563]: W0726 16:39:09.969798    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfefb5476_d140_46e0_87cf_128f5d761535.slice/crio-048e00fa1de27bae7b21d1941a482d08f17a1cb966c117c0676bad1dcdf29c32 WatchSource:0}: Error finding container 048e00fa1de27bae7b21d1941a482d08f17a1cb966c117c0676bad1dcdf29c32: Status 404 returned error can't find the container with id 048e00fa1de27bae7b21d1941a482d08f17a1cb966c117c0676bad1dcdf29c32
Jul 26 16:39:09.971267 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:09.971246    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 26 16:39:09.973155 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:09.973130    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:39:10.095714 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:10.095633    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:39:10.198864 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:10.198830    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:39:10.232159 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:10.229826    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bdfbca-5686c86bd4-csh8l"]
Jul 26 16:39:10.239815 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:10.239789    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bdfbca-5686c86bd4-csh8l"
Jul 26 16:39:10.243013 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:10.242993    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-bdfbca\""
Jul 26 16:39:10.248304 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:10.248284    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bdfbca-5686c86bd4-csh8l"]
Jul 26 16:39:10.251296 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:10.251271    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bdfbca-5bfc8ffcd6-cwmtd"]
Jul 26 16:39:10.255098 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:10.255081    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bdfbca-5bfc8ffcd6-cwmtd"
Jul 26 16:39:10.259225 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:10.259206    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-bdfbca-dockercfg-6rp2c\""
Jul 26 16:39:10.263046 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:10.263023    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bdfbca-5bfc8ffcd6-cwmtd"]
Jul 26 16:39:10.334233 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:10.334182    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9eab8b8-14ad-400e-9cb7-8a3a34ad3601-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bdfbca-5686c86bd4-csh8l\" (UID: \"f9eab8b8-14ad-400e-9cb7-8a3a34ad3601\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bdfbca-5686c86bd4-csh8l"
Jul 26 16:39:10.334445 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:10.334268    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4974c058-344e-4efc-8b4f-f33a05709acd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-bdfbca-5bfc8ffcd6-cwmtd\" (UID: \"4974c058-344e-4efc-8b4f-f33a05709acd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bdfbca-5bfc8ffcd6-cwmtd"
Jul 26 16:39:10.334445 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:10.334299    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6d47\" (UniqueName: \"kubernetes.io/projected/4974c058-344e-4efc-8b4f-f33a05709acd-kube-api-access-f6d47\") pod \"payload-processing-e2e-ipp-a-bdfbca-5bfc8ffcd6-cwmtd\" (UID: \"4974c058-344e-4efc-8b4f-f33a05709acd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bdfbca-5bfc8ffcd6-cwmtd"
Jul 26 16:39:10.334445 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:10.334326    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddpqz\" (UniqueName: \"kubernetes.io/projected/f9eab8b8-14ad-400e-9cb7-8a3a34ad3601-kube-api-access-ddpqz\") pod \"payload-pre-processing-e2e-ipp-a-bdfbca-5686c86bd4-csh8l\" (UID: \"f9eab8b8-14ad-400e-9cb7-8a3a34ad3601\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bdfbca-5686c86bd4-csh8l"
Jul 26 16:39:10.435006 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:10.434930    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9eab8b8-14ad-400e-9cb7-8a3a34ad3601-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bdfbca-5686c86bd4-csh8l\" (UID: \"f9eab8b8-14ad-400e-9cb7-8a3a34ad3601\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bdfbca-5686c86bd4-csh8l"
Jul 26 16:39:10.435006 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:10.434977    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4974c058-344e-4efc-8b4f-f33a05709acd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-bdfbca-5bfc8ffcd6-cwmtd\" (UID: \"4974c058-344e-4efc-8b4f-f33a05709acd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bdfbca-5bfc8ffcd6-cwmtd"
Jul 26 16:39:10.435006 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:10.435000    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6d47\" (UniqueName: \"kubernetes.io/projected/4974c058-344e-4efc-8b4f-f33a05709acd-kube-api-access-f6d47\") pod \"payload-processing-e2e-ipp-a-bdfbca-5bfc8ffcd6-cwmtd\" (UID: \"4974c058-344e-4efc-8b4f-f33a05709acd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bdfbca-5bfc8ffcd6-cwmtd"
Jul 26 16:39:10.435233 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:10.435021    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddpqz\" (UniqueName: \"kubernetes.io/projected/f9eab8b8-14ad-400e-9cb7-8a3a34ad3601-kube-api-access-ddpqz\") pod \"payload-pre-processing-e2e-ipp-a-bdfbca-5686c86bd4-csh8l\" (UID: \"f9eab8b8-14ad-400e-9cb7-8a3a34ad3601\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bdfbca-5686c86bd4-csh8l"
Jul 26 16:39:10.435667 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:10.435647    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4974c058-344e-4efc-8b4f-f33a05709acd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-bdfbca-5bfc8ffcd6-cwmtd\" (UID: \"4974c058-344e-4efc-8b4f-f33a05709acd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bdfbca-5bfc8ffcd6-cwmtd"
Jul 26 16:39:10.435974 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:10.435951    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9eab8b8-14ad-400e-9cb7-8a3a34ad3601-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bdfbca-5686c86bd4-csh8l\" (UID: \"f9eab8b8-14ad-400e-9cb7-8a3a34ad3601\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bdfbca-5686c86bd4-csh8l"
Jul 26 16:39:10.445150 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:10.445121    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6d47\" (UniqueName: \"kubernetes.io/projected/4974c058-344e-4efc-8b4f-f33a05709acd-kube-api-access-f6d47\") pod \"payload-processing-e2e-ipp-a-bdfbca-5bfc8ffcd6-cwmtd\" (UID: \"4974c058-344e-4efc-8b4f-f33a05709acd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bdfbca-5bfc8ffcd6-cwmtd"
Jul 26 16:39:10.445375 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:10.445355    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddpqz\" (UniqueName: \"kubernetes.io/projected/f9eab8b8-14ad-400e-9cb7-8a3a34ad3601-kube-api-access-ddpqz\") pod \"payload-pre-processing-e2e-ipp-a-bdfbca-5686c86bd4-csh8l\" (UID: \"f9eab8b8-14ad-400e-9cb7-8a3a34ad3601\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bdfbca-5686c86bd4-csh8l"
Jul 26 16:39:10.553377 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:10.553350    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bdfbca-5686c86bd4-csh8l"
Jul 26 16:39:10.567030 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:10.567010    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bdfbca-5bfc8ffcd6-cwmtd"
Jul 26 16:39:10.718024 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:10.717634    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bdfbca-5686c86bd4-csh8l"]
Jul 26 16:39:10.722540 ip-10-0-133-204 kubenswrapper[2563]: W0726 16:39:10.722251    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9eab8b8_14ad_400e_9cb7_8a3a34ad3601.slice/crio-aa7be22ae69f16d8fd5120370410d8a41c893799b0836d4190dc3ac8347eacb7 WatchSource:0}: Error finding container aa7be22ae69f16d8fd5120370410d8a41c893799b0836d4190dc3ac8347eacb7: Status 404 returned error can't find the container with id aa7be22ae69f16d8fd5120370410d8a41c893799b0836d4190dc3ac8347eacb7
Jul 26 16:39:10.752148 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:10.752120    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bdfbca-5bfc8ffcd6-cwmtd"]
Jul 26 16:39:10.754986 ip-10-0-133-204 kubenswrapper[2563]: W0726 16:39:10.754962    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4974c058_344e_4efc_8b4f_f33a05709acd.slice/crio-794cea73091bc470b017d35710a0a4e14ccc50d849eff8f8a6bdd95624767f4a WatchSource:0}: Error finding container 794cea73091bc470b017d35710a0a4e14ccc50d849eff8f8a6bdd95624767f4a: Status 404 returned error can't find the container with id 794cea73091bc470b017d35710a0a4e14ccc50d849eff8f8a6bdd95624767f4a
Jul 26 16:39:10.912848 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:10.912807    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bdfbca-5686c86bd4-csh8l" event={"ID":"f9eab8b8-14ad-400e-9cb7-8a3a34ad3601","Type":"ContainerStarted","Data":"8d62f36bd24c3de142ee3f6943cfcadc2b99d504cd98619468bfd44b3980c1d4"}
Jul 26 16:39:10.913058 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:10.912882    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bdfbca-5686c86bd4-csh8l" event={"ID":"f9eab8b8-14ad-400e-9cb7-8a3a34ad3601","Type":"ContainerStarted","Data":"aa7be22ae69f16d8fd5120370410d8a41c893799b0836d4190dc3ac8347eacb7"}
Jul 26 16:39:10.913058 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:10.912918    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bdfbca-5686c86bd4-csh8l"
Jul 26 16:39:10.914584 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:10.914554    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bdfbca-5bfc8ffcd6-cwmtd" event={"ID":"4974c058-344e-4efc-8b4f-f33a05709acd","Type":"ContainerStarted","Data":"2bf9ec4ecd66230932fc8e5daa6cbec17e95f41debee7da2ff39c552fd806552"}
Jul 26 16:39:10.914727 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:10.914606    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bdfbca-5bfc8ffcd6-cwmtd" event={"ID":"4974c058-344e-4efc-8b4f-f33a05709acd","Type":"ContainerStarted","Data":"794cea73091bc470b017d35710a0a4e14ccc50d849eff8f8a6bdd95624767f4a"}
Jul 26 16:39:10.914727 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:10.914633    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-bdfbca-5bfc8ffcd6-cwmtd"
Jul 26 16:39:10.916068 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:10.916046    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bdfbca-5d9d85bd8b-7cl5p" event={"ID":"fefb5476-d140-46e0-87cf-128f5d761535","Type":"ContainerStarted","Data":"3db326f9070d9a225a2bb4123e39b8b9e3ae5d68d5ee79ed304e62c3d92621fa"}
Jul 26 16:39:10.916179 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:10.916083    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bdfbca-5d9d85bd8b-7cl5p" event={"ID":"fefb5476-d140-46e0-87cf-128f5d761535","Type":"ContainerStarted","Data":"048e00fa1de27bae7b21d1941a482d08f17a1cb966c117c0676bad1dcdf29c32"}
Jul 26 16:39:10.916179 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:10.916102    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bdfbca-5d9d85bd8b-7cl5p"
Jul 26 16:39:10.931347 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:10.931286    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bdfbca-5686c86bd4-csh8l" podStartSLOduration=0.931265064 podStartE2EDuration="931.265064ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:39:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:39:10.930412068 +0000 UTC m=+2943.034637907" watchObservedRunningTime="2026-07-26 16:39:10.931265064 +0000 UTC m=+2943.035490874"
Jul 26 16:39:10.948631 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:10.948583    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-bdfbca-5bfc8ffcd6-cwmtd" podStartSLOduration=0.948566597 podStartE2EDuration="948.566597ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:39:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:39:10.946004887 +0000 UTC m=+2943.050230695" watchObservedRunningTime="2026-07-26 16:39:10.948566597 +0000 UTC m=+2943.052792451"
Jul 26 16:39:10.962680 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:10.962633    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bdfbca-5d9d85bd8b-7cl5p" podStartSLOduration=1.433416368 podStartE2EDuration="1.962621296s" totalImagesPullingTime="529.204928ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:39:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:39:10.961691286 +0000 UTC m=+2943.065917097" watchObservedRunningTime="2026-07-26 16:39:10.962621296 +0000 UTC m=+2943.066847102"
Jul 26 16:39:11.412183 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:11.412149    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:39:16.927773 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:16.927740    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bdfbca-5d9d85bd8b-7cl5p"
Jul 26 16:39:21.924442 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:21.924405    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-bdfbca-5bfc8ffcd6-cwmtd"
Jul 26 16:39:21.924968 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:21.924611    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bdfbca-5686c86bd4-csh8l"
Jul 26 16:39:27.470074 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:27.470039    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:39:27.913867 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:27.913835    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:39:28.100403 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:28.100361    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:39:30.760191 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:30.760088    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:39:30.861968 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:30.861929    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:39:32.559138 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:32.559102    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-33839d-58688b59b9-78zdw"]
Jul 26 16:39:32.563517 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:32.563493    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-33839d-58688b59b9-78zdw"
Jul 26 16:39:32.566685 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:32.566512    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-33839d-serving-cert\""
Jul 26 16:39:32.579754 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:32.579720    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-33839d-58688b59b9-78zdw"]
Jul 26 16:39:32.606531 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:32.606501    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:39:32.647032 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:32.647002    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w44tm\" (UniqueName: \"kubernetes.io/projected/779bfac4-792d-476b-8c4c-82638d8f53c5-kube-api-access-w44tm\") pod \"maas-api-e2e-ipp-b-33839d-58688b59b9-78zdw\" (UID: \"779bfac4-792d-476b-8c4c-82638d8f53c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-33839d-58688b59b9-78zdw"
Jul 26 16:39:32.647232 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:32.647049    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/779bfac4-792d-476b-8c4c-82638d8f53c5-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-33839d-58688b59b9-78zdw\" (UID: \"779bfac4-792d-476b-8c4c-82638d8f53c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-33839d-58688b59b9-78zdw"
Jul 26 16:39:32.647232 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:32.647078    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/779bfac4-792d-476b-8c4c-82638d8f53c5-maas-api-tls\") pod \"maas-api-e2e-ipp-b-33839d-58688b59b9-78zdw\" (UID: \"779bfac4-792d-476b-8c4c-82638d8f53c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-33839d-58688b59b9-78zdw"
Jul 26 16:39:32.747688 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:32.747651    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w44tm\" (UniqueName: \"kubernetes.io/projected/779bfac4-792d-476b-8c4c-82638d8f53c5-kube-api-access-w44tm\") pod \"maas-api-e2e-ipp-b-33839d-58688b59b9-78zdw\" (UID: \"779bfac4-792d-476b-8c4c-82638d8f53c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-33839d-58688b59b9-78zdw"
Jul 26 16:39:32.747926 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:32.747703    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/779bfac4-792d-476b-8c4c-82638d8f53c5-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-33839d-58688b59b9-78zdw\" (UID: \"779bfac4-792d-476b-8c4c-82638d8f53c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-33839d-58688b59b9-78zdw"
Jul 26 16:39:32.747926 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:32.747821    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/779bfac4-792d-476b-8c4c-82638d8f53c5-maas-api-tls\") pod \"maas-api-e2e-ipp-b-33839d-58688b59b9-78zdw\" (UID: \"779bfac4-792d-476b-8c4c-82638d8f53c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-33839d-58688b59b9-78zdw"
Jul 26 16:39:32.748052 ip-10-0-133-204 kubenswrapper[2563]: E0726 16:39:32.747983    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-33839d-serving-cert: secret "maas-api-e2e-ipp-b-33839d-serving-cert" not found
Jul 26 16:39:32.748052 ip-10-0-133-204 kubenswrapper[2563]: E0726 16:39:32.748045    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/779bfac4-792d-476b-8c4c-82638d8f53c5-maas-api-tls podName:779bfac4-792d-476b-8c4c-82638d8f53c5 nodeName:}" failed. No retries permitted until 2026-07-26 16:39:33.248028199 +0000 UTC m=+2965.352253988 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/779bfac4-792d-476b-8c4c-82638d8f53c5-maas-api-tls") pod "maas-api-e2e-ipp-b-33839d-58688b59b9-78zdw" (UID: "779bfac4-792d-476b-8c4c-82638d8f53c5") : secret "maas-api-e2e-ipp-b-33839d-serving-cert" not found
Jul 26 16:39:32.748377 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:32.748360    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/779bfac4-792d-476b-8c4c-82638d8f53c5-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-33839d-58688b59b9-78zdw\" (UID: \"779bfac4-792d-476b-8c4c-82638d8f53c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-33839d-58688b59b9-78zdw"
Jul 26 16:39:32.756527 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:32.756492    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w44tm\" (UniqueName: \"kubernetes.io/projected/779bfac4-792d-476b-8c4c-82638d8f53c5-kube-api-access-w44tm\") pod \"maas-api-e2e-ipp-b-33839d-58688b59b9-78zdw\" (UID: \"779bfac4-792d-476b-8c4c-82638d8f53c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-33839d-58688b59b9-78zdw"
Jul 26 16:39:32.834162 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:32.834065    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:39:33.135445 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:33.135364    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:39:33.238296 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:33.238262    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:39:33.240839 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:33.240809    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-33839d-6b4fbc6664-sr92w"]
Jul 26 16:39:33.246141 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:33.246115    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-33839d-6b4fbc6664-sr92w"
Jul 26 16:39:33.248960 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:33.248934    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-33839d-dockercfg-dth5h\""
Jul 26 16:39:33.249085 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:33.249060    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-33839d\""
Jul 26 16:39:33.254223 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:33.253869    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/779bfac4-792d-476b-8c4c-82638d8f53c5-maas-api-tls\") pod \"maas-api-e2e-ipp-b-33839d-58688b59b9-78zdw\" (UID: \"779bfac4-792d-476b-8c4c-82638d8f53c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-33839d-58688b59b9-78zdw"
Jul 26 16:39:33.256894 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:33.256868    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/779bfac4-792d-476b-8c4c-82638d8f53c5-maas-api-tls\") pod \"maas-api-e2e-ipp-b-33839d-58688b59b9-78zdw\" (UID: \"779bfac4-792d-476b-8c4c-82638d8f53c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-33839d-58688b59b9-78zdw"
Jul 26 16:39:33.257252 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:33.257227    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-33839d-6b4fbc6664-sr92w"]
Jul 26 16:39:33.355465 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:33.355430    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29c9l\" (UniqueName: \"kubernetes.io/projected/7901a20c-246a-42c4-a6d8-9fcaebb6edca-kube-api-access-29c9l\") pod \"payload-pre-processing-e2e-ipp-b-33839d-6b4fbc6664-sr92w\" (UID: \"7901a20c-246a-42c4-a6d8-9fcaebb6edca\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-33839d-6b4fbc6664-sr92w"
Jul 26 16:39:33.355465 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:33.355472    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7901a20c-246a-42c4-a6d8-9fcaebb6edca-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-33839d-6b4fbc6664-sr92w\" (UID: \"7901a20c-246a-42c4-a6d8-9fcaebb6edca\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-33839d-6b4fbc6664-sr92w"
Jul 26 16:39:33.456247 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:33.456160    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29c9l\" (UniqueName: \"kubernetes.io/projected/7901a20c-246a-42c4-a6d8-9fcaebb6edca-kube-api-access-29c9l\") pod \"payload-pre-processing-e2e-ipp-b-33839d-6b4fbc6664-sr92w\" (UID: \"7901a20c-246a-42c4-a6d8-9fcaebb6edca\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-33839d-6b4fbc6664-sr92w"
Jul 26 16:39:33.456247 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:33.456203    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7901a20c-246a-42c4-a6d8-9fcaebb6edca-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-33839d-6b4fbc6664-sr92w\" (UID: \"7901a20c-246a-42c4-a6d8-9fcaebb6edca\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-33839d-6b4fbc6664-sr92w"
Jul 26 16:39:33.456993 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:33.456973    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7901a20c-246a-42c4-a6d8-9fcaebb6edca-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-33839d-6b4fbc6664-sr92w\" (UID: \"7901a20c-246a-42c4-a6d8-9fcaebb6edca\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-33839d-6b4fbc6664-sr92w"
Jul 26 16:39:33.468096 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:33.468074    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29c9l\" (UniqueName: \"kubernetes.io/projected/7901a20c-246a-42c4-a6d8-9fcaebb6edca-kube-api-access-29c9l\") pod \"payload-pre-processing-e2e-ipp-b-33839d-6b4fbc6664-sr92w\" (UID: \"7901a20c-246a-42c4-a6d8-9fcaebb6edca\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-33839d-6b4fbc6664-sr92w"
Jul 26 16:39:33.476915 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:33.476896    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-33839d-58688b59b9-78zdw"
Jul 26 16:39:33.559346 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:33.559262    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-33839d-6b4fbc6664-sr92w"
Jul 26 16:39:33.626916 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:33.626750    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-33839d-58688b59b9-78zdw"]
Jul 26 16:39:33.629216 ip-10-0-133-204 kubenswrapper[2563]: W0726 16:39:33.629171    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod779bfac4_792d_476b_8c4c_82638d8f53c5.slice/crio-ec1284df3087f1d09bb7a12485af96c24331640a5f485cb2022df13a3833d0de WatchSource:0}: Error finding container ec1284df3087f1d09bb7a12485af96c24331640a5f485cb2022df13a3833d0de: Status 404 returned error can't find the container with id ec1284df3087f1d09bb7a12485af96c24331640a5f485cb2022df13a3833d0de
Jul 26 16:39:33.917975 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:33.917949    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-33839d-6b4fbc6664-sr92w"]
Jul 26 16:39:33.920133 ip-10-0-133-204 kubenswrapper[2563]: W0726 16:39:33.920101    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7901a20c_246a_42c4_a6d8_9fcaebb6edca.slice/crio-aad1869127e210e04e4a96016fd60dff3fe0338c1b3f4fb19898388c09338dbd WatchSource:0}: Error finding container aad1869127e210e04e4a96016fd60dff3fe0338c1b3f4fb19898388c09338dbd: Status 404 returned error can't find the container with id aad1869127e210e04e4a96016fd60dff3fe0338c1b3f4fb19898388c09338dbd
Jul 26 16:39:34.054386 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:34.054360    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-33839d-6b4fbc6664-sr92w" event={"ID":"7901a20c-246a-42c4-a6d8-9fcaebb6edca","Type":"ContainerStarted","Data":"3b11dbf980b63cbcbafb81c6ed11be54de918fc77e72f16751eb2220f20183fe"}
Jul 26 16:39:34.054543 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:34.054426    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-33839d-6b4fbc6664-sr92w"
Jul 26 16:39:34.054543 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:34.054453    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-33839d-6b4fbc6664-sr92w" event={"ID":"7901a20c-246a-42c4-a6d8-9fcaebb6edca","Type":"ContainerStarted","Data":"aad1869127e210e04e4a96016fd60dff3fe0338c1b3f4fb19898388c09338dbd"}
Jul 26 16:39:34.055550 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:34.055530    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-33839d-58688b59b9-78zdw" event={"ID":"779bfac4-792d-476b-8c4c-82638d8f53c5","Type":"ContainerStarted","Data":"ec1284df3087f1d09bb7a12485af96c24331640a5f485cb2022df13a3833d0de"}
Jul 26 16:39:34.071070 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:34.071031    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-33839d-6b4fbc6664-sr92w" podStartSLOduration=1.071016664 podStartE2EDuration="1.071016664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:39:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:39:34.070318906 +0000 UTC m=+2966.174544715" watchObservedRunningTime="2026-07-26 16:39:34.071016664 +0000 UTC m=+2966.175242471"
Jul 26 16:39:35.062643 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:35.062600    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-33839d-58688b59b9-78zdw" event={"ID":"779bfac4-792d-476b-8c4c-82638d8f53c5","Type":"ContainerStarted","Data":"a34e9fcdb93ee4a7f8141620eceedaa7c64d3bcd5406a9eae1c26718907e1332"}
Jul 26 16:39:35.063114 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:35.062816    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-33839d-58688b59b9-78zdw"
Jul 26 16:39:35.083818 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:35.083745    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-33839d-58688b59b9-78zdw" podStartSLOduration=2.676629363 podStartE2EDuration="3.083730572s" totalImagesPullingTime="407.101209ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:39:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:39:35.082155716 +0000 UTC m=+2967.186381522" watchObservedRunningTime="2026-07-26 16:39:35.083730572 +0000 UTC m=+2967.187956442"
Jul 26 16:39:35.250916 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:35.250885    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:39:41.073584 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:41.073555    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-33839d-58688b59b9-78zdw"
Jul 26 16:39:45.065288 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:45.065251    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-33839d-6b4fbc6664-sr92w"
Jul 26 16:39:57.711105 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:39:57.711076    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-rc9pz_9c08ac5a-85ff-4504-9d77-b8f41be0242a/payload-processing/0.log"
Jul 26 16:40:21.748396 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:40:21.748346    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:41:05.272662 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:41:05.272577    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-bdfbca-5bfc8ffcd6-cwmtd_4974c058-344e-4efc-8b4f-f33a05709acd/payload-processing/0.log"
Jul 26 16:41:05.400267 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:41:05.400244    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-rc9pz_9c08ac5a-85ff-4504-9d77-b8f41be0242a/payload-processing/0.log"
Jul 26 16:41:05.851108 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:41:05.851074    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:41:06.140569 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:41:06.140464    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:41:06.537895 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:41:06.537861    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:41:09.423862 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:41:09.423826    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:41:09.527393 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:41:09.527346    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:41:12.059258 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:41:12.059219    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3e477e-7b99c77f48-8wmdl"]
Jul 26 16:41:12.063749 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:41:12.063726    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3e477e-7b99c77f48-8wmdl"
Jul 26 16:41:12.066090 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:41:12.066064    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-3e477e-serving-cert\""
Jul 26 16:41:12.071948 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:41:12.071922    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3e477e-7b99c77f48-8wmdl"]
Jul 26 16:41:12.091076 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:41:12.091048    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:41:12.118471 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:41:12.118440    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20faa6e9-3a39-422e-9742-ef1360793e79-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-3e477e-7b99c77f48-8wmdl\" (UID: \"20faa6e9-3a39-422e-9742-ef1360793e79\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3e477e-7b99c77f48-8wmdl"
Jul 26 16:41:12.118609 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:41:12.118487    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk598\" (UniqueName: \"kubernetes.io/projected/20faa6e9-3a39-422e-9742-ef1360793e79-kube-api-access-fk598\") pod \"maas-api-e2e-ipp-cleanup-3e477e-7b99c77f48-8wmdl\" (UID: \"20faa6e9-3a39-422e-9742-ef1360793e79\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3e477e-7b99c77f48-8wmdl"
Jul 26 16:41:12.118609 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:41:12.118546    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20faa6e9-3a39-422e-9742-ef1360793e79-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-3e477e-7b99c77f48-8wmdl\" (UID: \"20faa6e9-3a39-422e-9742-ef1360793e79\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3e477e-7b99c77f48-8wmdl"
Jul 26 16:41:12.219119 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:41:12.219077    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20faa6e9-3a39-422e-9742-ef1360793e79-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-3e477e-7b99c77f48-8wmdl\" (UID: \"20faa6e9-3a39-422e-9742-ef1360793e79\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3e477e-7b99c77f48-8wmdl"
Jul 26 16:41:12.219300 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:41:12.219151    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk598\" (UniqueName: \"kubernetes.io/projected/20faa6e9-3a39-422e-9742-ef1360793e79-kube-api-access-fk598\") pod \"maas-api-e2e-ipp-cleanup-3e477e-7b99c77f48-8wmdl\" (UID: \"20faa6e9-3a39-422e-9742-ef1360793e79\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3e477e-7b99c77f48-8wmdl"
Jul 26 16:41:12.219300 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:41:12.219210    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20faa6e9-3a39-422e-9742-ef1360793e79-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-3e477e-7b99c77f48-8wmdl\" (UID: \"20faa6e9-3a39-422e-9742-ef1360793e79\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3e477e-7b99c77f48-8wmdl"
Jul 26 16:41:12.219375 ip-10-0-133-204 kubenswrapper[2563]: E0726 16:41:12.219362    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3e477e-serving-cert: secret "maas-api-e2e-ipp-cleanup-3e477e-serving-cert" not found
Jul 26 16:41:12.219471 ip-10-0-133-204 kubenswrapper[2563]: E0726 16:41:12.219459    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20faa6e9-3a39-422e-9742-ef1360793e79-maas-api-tls podName:20faa6e9-3a39-422e-9742-ef1360793e79 nodeName:}" failed. No retries permitted until 2026-07-26 16:41:12.719437006 +0000 UTC m=+3064.823662807 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/20faa6e9-3a39-422e-9742-ef1360793e79-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-3e477e-7b99c77f48-8wmdl" (UID: "20faa6e9-3a39-422e-9742-ef1360793e79") : secret "maas-api-e2e-ipp-cleanup-3e477e-serving-cert" not found
Jul 26 16:41:12.219836 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:41:12.219819    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20faa6e9-3a39-422e-9742-ef1360793e79-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-3e477e-7b99c77f48-8wmdl\" (UID: \"20faa6e9-3a39-422e-9742-ef1360793e79\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3e477e-7b99c77f48-8wmdl"
Jul 26 16:41:12.230217 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:41:12.230179    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk598\" (UniqueName: \"kubernetes.io/projected/20faa6e9-3a39-422e-9742-ef1360793e79-kube-api-access-fk598\") pod \"maas-api-e2e-ipp-cleanup-3e477e-7b99c77f48-8wmdl\" (UID: \"20faa6e9-3a39-422e-9742-ef1360793e79\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3e477e-7b99c77f48-8wmdl"
Jul 26 16:41:12.414838 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:41:12.414727    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:41:12.607859 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:41:12.607826    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:41:12.706577 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:41:12.706497    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:41:12.724995 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:41:12.724967    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20faa6e9-3a39-422e-9742-ef1360793e79-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-3e477e-7b99c77f48-8wmdl\" (UID: \"20faa6e9-3a39-422e-9742-ef1360793e79\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3e477e-7b99c77f48-8wmdl"
Jul 26 16:41:12.727653 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:41:12.727629    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20faa6e9-3a39-422e-9742-ef1360793e79-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-3e477e-7b99c77f48-8wmdl\" (UID: \"20faa6e9-3a39-422e-9742-ef1360793e79\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3e477e-7b99c77f48-8wmdl"
Jul 26 16:41:12.976865 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:41:12.976837    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3e477e-7b99c77f48-8wmdl"
Jul 26 16:41:13.334039 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:41:13.334013    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3e477e-7b99c77f48-8wmdl"]
Jul 26 16:41:13.335519 ip-10-0-133-204 kubenswrapper[2563]: W0726 16:41:13.335496    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20faa6e9_3a39_422e_9742_ef1360793e79.slice/crio-0ecb8ce2940848b5827006203d0c9bf3cdd1881f069ebb822b65f0dc65e14d42 WatchSource:0}: Error finding container 0ecb8ce2940848b5827006203d0c9bf3cdd1881f069ebb822b65f0dc65e14d42: Status 404 returned error can't find the container with id 0ecb8ce2940848b5827006203d0c9bf3cdd1881f069ebb822b65f0dc65e14d42
Jul 26 16:41:13.578913 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:41:13.578877    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3e477e-7b99c77f48-8wmdl" event={"ID":"20faa6e9-3a39-422e-9742-ef1360793e79","Type":"ContainerStarted","Data":"0ecb8ce2940848b5827006203d0c9bf3cdd1881f069ebb822b65f0dc65e14d42"}
Jul 26 16:41:13.905795 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:41:13.905734    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:41:14.585944 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:41:14.585913    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3e477e-7b99c77f48-8wmdl" event={"ID":"20faa6e9-3a39-422e-9742-ef1360793e79","Type":"ContainerStarted","Data":"a9c2c8d6bebf473ea64e20952931cf8bd158bccca64646f0162b9d19c8c01afa"}
Jul 26 16:41:14.586361 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:41:14.585986    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3e477e-7b99c77f48-8wmdl"
Jul 26 16:41:14.605746 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:41:14.605694    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3e477e-7b99c77f48-8wmdl" podStartSLOduration=2.1797574490000002 podStartE2EDuration="2.605676924s" totalImagesPullingTime="425.919475ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:41:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:41:14.603936384 +0000 UTC m=+3066.708162195" watchObservedRunningTime="2026-07-26 16:41:14.605676924 +0000 UTC m=+3066.709902791"
Jul 26 16:41:20.597300 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:41:20.597268    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3e477e-7b99c77f48-8wmdl"
Jul 26 16:41:28.504932 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:41:28.504895    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:41:28.629958 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:41:28.629932    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:42:01.612256 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:01.612218    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3e477e-7b99c77f48-8wmdl"]
Jul 26 16:42:01.612934 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:01.612502    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3e477e-7b99c77f48-8wmdl" podUID="20faa6e9-3a39-422e-9742-ef1360793e79" containerName="maas-api" containerID="cri-o://a9c2c8d6bebf473ea64e20952931cf8bd158bccca64646f0162b9d19c8c01afa" gracePeriod=30
Jul 26 16:42:01.644488 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:01.644458    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:42:01.726383 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:01.726350    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:42:01.797258 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:01.797204    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:42:01.838144 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:01.838108    2563 generic.go:356] "Generic (PLEG): container finished" podID="20faa6e9-3a39-422e-9742-ef1360793e79" containerID="a9c2c8d6bebf473ea64e20952931cf8bd158bccca64646f0162b9d19c8c01afa" exitCode=0
Jul 26 16:42:01.838321 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:01.838185    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3e477e-7b99c77f48-8wmdl" event={"ID":"20faa6e9-3a39-422e-9742-ef1360793e79","Type":"ContainerDied","Data":"a9c2c8d6bebf473ea64e20952931cf8bd158bccca64646f0162b9d19c8c01afa"}
Jul 26 16:42:01.871002 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:01.870945    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3e477e-7b99c77f48-8wmdl"
Jul 26 16:42:01.914491 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:01.914465    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fk598\" (UniqueName: \"kubernetes.io/projected/20faa6e9-3a39-422e-9742-ef1360793e79-kube-api-access-fk598\") pod \"20faa6e9-3a39-422e-9742-ef1360793e79\" (UID: \"20faa6e9-3a39-422e-9742-ef1360793e79\") "
Jul 26 16:42:01.914679 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:01.914523    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20faa6e9-3a39-422e-9742-ef1360793e79-maas-api-tls\") pod \"20faa6e9-3a39-422e-9742-ef1360793e79\" (UID: \"20faa6e9-3a39-422e-9742-ef1360793e79\") "
Jul 26 16:42:01.914679 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:01.914596    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20faa6e9-3a39-422e-9742-ef1360793e79-openshift-service-ca\") pod \"20faa6e9-3a39-422e-9742-ef1360793e79\" (UID: \"20faa6e9-3a39-422e-9742-ef1360793e79\") "
Jul 26 16:42:01.915132 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:01.914925    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20faa6e9-3a39-422e-9742-ef1360793e79-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "20faa6e9-3a39-422e-9742-ef1360793e79" (UID: "20faa6e9-3a39-422e-9742-ef1360793e79"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 26 16:42:01.916935 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:01.916905    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20faa6e9-3a39-422e-9742-ef1360793e79-kube-api-access-fk598" (OuterVolumeSpecName: "kube-api-access-fk598") pod "20faa6e9-3a39-422e-9742-ef1360793e79" (UID: "20faa6e9-3a39-422e-9742-ef1360793e79"). InnerVolumeSpecName "kube-api-access-fk598". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 26 16:42:01.917045 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:01.916964    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20faa6e9-3a39-422e-9742-ef1360793e79-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "20faa6e9-3a39-422e-9742-ef1360793e79" (UID: "20faa6e9-3a39-422e-9742-ef1360793e79"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 26 16:42:02.016194 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:02.016163    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20faa6e9-3a39-422e-9742-ef1360793e79-openshift-service-ca\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:42:02.016194 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:02.016188    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fk598\" (UniqueName: \"kubernetes.io/projected/20faa6e9-3a39-422e-9742-ef1360793e79-kube-api-access-fk598\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:42:02.016194 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:02.016199    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20faa6e9-3a39-422e-9742-ef1360793e79-maas-api-tls\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:42:02.845119 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:02.845080    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3e477e-7b99c77f48-8wmdl" event={"ID":"20faa6e9-3a39-422e-9742-ef1360793e79","Type":"ContainerDied","Data":"0ecb8ce2940848b5827006203d0c9bf3cdd1881f069ebb822b65f0dc65e14d42"}
Jul 26 16:42:02.845579 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:02.845125    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3e477e-7b99c77f48-8wmdl"
Jul 26 16:42:02.845579 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:02.845181    2563 scope.go:117] "RemoveContainer" containerID="a9c2c8d6bebf473ea64e20952931cf8bd158bccca64646f0162b9d19c8c01afa"
Jul 26 16:42:02.867459 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:02.867433    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3e477e-7b99c77f48-8wmdl"]
Jul 26 16:42:02.872615 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:02.872587    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3e477e-7b99c77f48-8wmdl"]
Jul 26 16:42:04.373414 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:04.373385    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20faa6e9-3a39-422e-9742-ef1360793e79" path="/var/lib/kubelet/pods/20faa6e9-3a39-422e-9742-ef1360793e79/volumes"
Jul 26 16:42:05.347319 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:05.347285    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:42:05.775343 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:05.775316    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:42:13.484345 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:13.484315    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:42:13.642756 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:13.642719    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:42:13.852065 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:13.851959    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:42:14.076329 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:14.076302    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:42:26.170160 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:26.170117    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:42:35.905902 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:35.905810    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bdfbca-5d9d85bd8b-7cl5p"]
Jul 26 16:42:35.906518 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:35.906225    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bdfbca-5d9d85bd8b-7cl5p" podUID="fefb5476-d140-46e0-87cf-128f5d761535" containerName="maas-api" containerID="cri-o://3db326f9070d9a225a2bb4123e39b8b9e3ae5d68d5ee79ed304e62c3d92621fa" gracePeriod=30
Jul 26 16:42:35.938040 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:35.938007    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:42:35.981144 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:35.981114    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bdfbca-5bfc8ffcd6-cwmtd"]
Jul 26 16:42:35.981350 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:35.981328    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-bdfbca-5bfc8ffcd6-cwmtd" podUID="4974c058-344e-4efc-8b4f-f33a05709acd" containerName="payload-processing" containerID="cri-o://2bf9ec4ecd66230932fc8e5daa6cbec17e95f41debee7da2ff39c552fd806552" gracePeriod=30
Jul 26 16:42:35.991736 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:35.991705    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bdfbca-5686c86bd4-csh8l"]
Jul 26 16:42:35.991981 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:35.991951    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bdfbca-5686c86bd4-csh8l" podUID="f9eab8b8-14ad-400e-9cb7-8a3a34ad3601" containerName="payload-pre-processing" containerID="cri-o://8d62f36bd24c3de142ee3f6943cfcadc2b99d504cd98619468bfd44b3980c1d4" gracePeriod=30
Jul 26 16:42:36.032161 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:36.032132    2563 generic.go:356] "Generic (PLEG): container finished" podID="fefb5476-d140-46e0-87cf-128f5d761535" containerID="3db326f9070d9a225a2bb4123e39b8b9e3ae5d68d5ee79ed304e62c3d92621fa" exitCode=0
Jul 26 16:42:36.032291 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:36.032201    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bdfbca-5d9d85bd8b-7cl5p" event={"ID":"fefb5476-d140-46e0-87cf-128f5d761535","Type":"ContainerDied","Data":"3db326f9070d9a225a2bb4123e39b8b9e3ae5d68d5ee79ed304e62c3d92621fa"}
Jul 26 16:42:36.047891 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:36.047859    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:42:36.117568 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:36.117425    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:42:36.314782 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:36.314740    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bdfbca-5686c86bd4-csh8l"
Jul 26 16:42:36.318021 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:36.317999    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bdfbca-5bfc8ffcd6-cwmtd"
Jul 26 16:42:36.332481 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:36.332457    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bdfbca-5d9d85bd8b-7cl5p"
Jul 26 16:42:36.356266 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:36.356241    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4974c058-344e-4efc-8b4f-f33a05709acd-plugins-config-volume\") pod \"4974c058-344e-4efc-8b4f-f33a05709acd\" (UID: \"4974c058-344e-4efc-8b4f-f33a05709acd\") "
Jul 26 16:42:36.356461 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:36.356275    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ddpqz\" (UniqueName: \"kubernetes.io/projected/f9eab8b8-14ad-400e-9cb7-8a3a34ad3601-kube-api-access-ddpqz\") pod \"f9eab8b8-14ad-400e-9cb7-8a3a34ad3601\" (UID: \"f9eab8b8-14ad-400e-9cb7-8a3a34ad3601\") "
Jul 26 16:42:36.356461 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:36.356306    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9eab8b8-14ad-400e-9cb7-8a3a34ad3601-plugins-config-volume\") pod \"f9eab8b8-14ad-400e-9cb7-8a3a34ad3601\" (UID: \"f9eab8b8-14ad-400e-9cb7-8a3a34ad3601\") "
Jul 26 16:42:36.356600 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:36.356461    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f6d47\" (UniqueName: \"kubernetes.io/projected/4974c058-344e-4efc-8b4f-f33a05709acd-kube-api-access-f6d47\") pod \"4974c058-344e-4efc-8b4f-f33a05709acd\" (UID: \"4974c058-344e-4efc-8b4f-f33a05709acd\") "
Jul 26 16:42:36.356683 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:36.356658    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4974c058-344e-4efc-8b4f-f33a05709acd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4974c058-344e-4efc-8b4f-f33a05709acd" (UID: "4974c058-344e-4efc-8b4f-f33a05709acd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 26 16:42:36.356831 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:36.356815    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4974c058-344e-4efc-8b4f-f33a05709acd-plugins-config-volume\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:42:36.356911 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:36.356837    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9eab8b8-14ad-400e-9cb7-8a3a34ad3601-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f9eab8b8-14ad-400e-9cb7-8a3a34ad3601" (UID: "f9eab8b8-14ad-400e-9cb7-8a3a34ad3601"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 26 16:42:36.358722 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:36.358699    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9eab8b8-14ad-400e-9cb7-8a3a34ad3601-kube-api-access-ddpqz" (OuterVolumeSpecName: "kube-api-access-ddpqz") pod "f9eab8b8-14ad-400e-9cb7-8a3a34ad3601" (UID: "f9eab8b8-14ad-400e-9cb7-8a3a34ad3601"). InnerVolumeSpecName "kube-api-access-ddpqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 26 16:42:36.358851 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:36.358820    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4974c058-344e-4efc-8b4f-f33a05709acd-kube-api-access-f6d47" (OuterVolumeSpecName: "kube-api-access-f6d47") pod "4974c058-344e-4efc-8b4f-f33a05709acd" (UID: "4974c058-344e-4efc-8b4f-f33a05709acd"). InnerVolumeSpecName "kube-api-access-f6d47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 26 16:42:36.458169 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:36.458079    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4tgv5\" (UniqueName: \"kubernetes.io/projected/fefb5476-d140-46e0-87cf-128f5d761535-kube-api-access-4tgv5\") pod \"fefb5476-d140-46e0-87cf-128f5d761535\" (UID: \"fefb5476-d140-46e0-87cf-128f5d761535\") "
Jul 26 16:42:36.458169 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:36.458166    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fefb5476-d140-46e0-87cf-128f5d761535-maas-api-tls\") pod \"fefb5476-d140-46e0-87cf-128f5d761535\" (UID: \"fefb5476-d140-46e0-87cf-128f5d761535\") "
Jul 26 16:42:36.458375 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:36.458280    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fefb5476-d140-46e0-87cf-128f5d761535-openshift-service-ca\") pod \"fefb5476-d140-46e0-87cf-128f5d761535\" (UID: \"fefb5476-d140-46e0-87cf-128f5d761535\") "
Jul 26 16:42:36.458527 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:36.458511    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6d47\" (UniqueName: \"kubernetes.io/projected/4974c058-344e-4efc-8b4f-f33a05709acd-kube-api-access-f6d47\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:42:36.458604 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:36.458529    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddpqz\" (UniqueName: \"kubernetes.io/projected/f9eab8b8-14ad-400e-9cb7-8a3a34ad3601-kube-api-access-ddpqz\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:42:36.458604 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:36.458544    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9eab8b8-14ad-400e-9cb7-8a3a34ad3601-plugins-config-volume\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:42:36.458709 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:36.458641    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fefb5476-d140-46e0-87cf-128f5d761535-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fefb5476-d140-46e0-87cf-128f5d761535" (UID: "fefb5476-d140-46e0-87cf-128f5d761535"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 26 16:42:36.460230 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:36.460208    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fefb5476-d140-46e0-87cf-128f5d761535-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fefb5476-d140-46e0-87cf-128f5d761535" (UID: "fefb5476-d140-46e0-87cf-128f5d761535"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 26 16:42:36.460308 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:36.460253    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fefb5476-d140-46e0-87cf-128f5d761535-kube-api-access-4tgv5" (OuterVolumeSpecName: "kube-api-access-4tgv5") pod "fefb5476-d140-46e0-87cf-128f5d761535" (UID: "fefb5476-d140-46e0-87cf-128f5d761535"). InnerVolumeSpecName "kube-api-access-4tgv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 26 16:42:36.559315 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:36.559275    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fefb5476-d140-46e0-87cf-128f5d761535-openshift-service-ca\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:42:36.559315 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:36.559305    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tgv5\" (UniqueName: \"kubernetes.io/projected/fefb5476-d140-46e0-87cf-128f5d761535-kube-api-access-4tgv5\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:42:36.559315 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:36.559318    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fefb5476-d140-46e0-87cf-128f5d761535-maas-api-tls\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:42:37.040007 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:37.039978    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bdfbca-5d9d85bd8b-7cl5p"
Jul 26 16:42:37.040007 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:37.040002    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bdfbca-5d9d85bd8b-7cl5p" event={"ID":"fefb5476-d140-46e0-87cf-128f5d761535","Type":"ContainerDied","Data":"048e00fa1de27bae7b21d1941a482d08f17a1cb966c117c0676bad1dcdf29c32"}
Jul 26 16:42:37.040494 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:37.040063    2563 scope.go:117] "RemoveContainer" containerID="3db326f9070d9a225a2bb4123e39b8b9e3ae5d68d5ee79ed304e62c3d92621fa"
Jul 26 16:42:37.041670 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:37.041636    2563 generic.go:356] "Generic (PLEG): container finished" podID="f9eab8b8-14ad-400e-9cb7-8a3a34ad3601" containerID="8d62f36bd24c3de142ee3f6943cfcadc2b99d504cd98619468bfd44b3980c1d4" exitCode=0
Jul 26 16:42:37.041794 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:37.041690    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bdfbca-5686c86bd4-csh8l"
Jul 26 16:42:37.041794 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:37.041724    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bdfbca-5686c86bd4-csh8l" event={"ID":"f9eab8b8-14ad-400e-9cb7-8a3a34ad3601","Type":"ContainerDied","Data":"8d62f36bd24c3de142ee3f6943cfcadc2b99d504cd98619468bfd44b3980c1d4"}
Jul 26 16:42:37.042018 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:37.041984    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bdfbca-5686c86bd4-csh8l" event={"ID":"f9eab8b8-14ad-400e-9cb7-8a3a34ad3601","Type":"ContainerDied","Data":"aa7be22ae69f16d8fd5120370410d8a41c893799b0836d4190dc3ac8347eacb7"}
Jul 26 16:42:37.043258 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:37.043233    2563 generic.go:356] "Generic (PLEG): container finished" podID="4974c058-344e-4efc-8b4f-f33a05709acd" containerID="2bf9ec4ecd66230932fc8e5daa6cbec17e95f41debee7da2ff39c552fd806552" exitCode=0
Jul 26 16:42:37.043371 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:37.043280    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bdfbca-5bfc8ffcd6-cwmtd"
Jul 26 16:42:37.043371 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:37.043293    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bdfbca-5bfc8ffcd6-cwmtd" event={"ID":"4974c058-344e-4efc-8b4f-f33a05709acd","Type":"ContainerDied","Data":"2bf9ec4ecd66230932fc8e5daa6cbec17e95f41debee7da2ff39c552fd806552"}
Jul 26 16:42:37.043371 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:37.043333    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bdfbca-5bfc8ffcd6-cwmtd" event={"ID":"4974c058-344e-4efc-8b4f-f33a05709acd","Type":"ContainerDied","Data":"794cea73091bc470b017d35710a0a4e14ccc50d849eff8f8a6bdd95624767f4a"}
Jul 26 16:42:37.050880 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:37.050866    2563 scope.go:117] "RemoveContainer" containerID="8d62f36bd24c3de142ee3f6943cfcadc2b99d504cd98619468bfd44b3980c1d4"
Jul 26 16:42:37.058777 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:37.058749    2563 scope.go:117] "RemoveContainer" containerID="8d62f36bd24c3de142ee3f6943cfcadc2b99d504cd98619468bfd44b3980c1d4"
Jul 26 16:42:37.059003 ip-10-0-133-204 kubenswrapper[2563]: E0726 16:42:37.058986    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d62f36bd24c3de142ee3f6943cfcadc2b99d504cd98619468bfd44b3980c1d4\": container with ID starting with 8d62f36bd24c3de142ee3f6943cfcadc2b99d504cd98619468bfd44b3980c1d4 not found: ID does not exist" containerID="8d62f36bd24c3de142ee3f6943cfcadc2b99d504cd98619468bfd44b3980c1d4"
Jul 26 16:42:37.059055 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:37.059011    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d62f36bd24c3de142ee3f6943cfcadc2b99d504cd98619468bfd44b3980c1d4"} err="failed to get container status \"8d62f36bd24c3de142ee3f6943cfcadc2b99d504cd98619468bfd44b3980c1d4\": rpc error: code = NotFound desc = could not find container \"8d62f36bd24c3de142ee3f6943cfcadc2b99d504cd98619468bfd44b3980c1d4\": container with ID starting with 8d62f36bd24c3de142ee3f6943cfcadc2b99d504cd98619468bfd44b3980c1d4 not found: ID does not exist"
Jul 26 16:42:37.059055 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:37.059027    2563 scope.go:117] "RemoveContainer" containerID="2bf9ec4ecd66230932fc8e5daa6cbec17e95f41debee7da2ff39c552fd806552"
Jul 26 16:42:37.067232 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:37.067215    2563 scope.go:117] "RemoveContainer" containerID="2bf9ec4ecd66230932fc8e5daa6cbec17e95f41debee7da2ff39c552fd806552"
Jul 26 16:42:37.067506 ip-10-0-133-204 kubenswrapper[2563]: E0726 16:42:37.067481    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bf9ec4ecd66230932fc8e5daa6cbec17e95f41debee7da2ff39c552fd806552\": container with ID starting with 2bf9ec4ecd66230932fc8e5daa6cbec17e95f41debee7da2ff39c552fd806552 not found: ID does not exist" containerID="2bf9ec4ecd66230932fc8e5daa6cbec17e95f41debee7da2ff39c552fd806552"
Jul 26 16:42:37.067581 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:37.067514    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bf9ec4ecd66230932fc8e5daa6cbec17e95f41debee7da2ff39c552fd806552"} err="failed to get container status \"2bf9ec4ecd66230932fc8e5daa6cbec17e95f41debee7da2ff39c552fd806552\": rpc error: code = NotFound desc = could not find container \"2bf9ec4ecd66230932fc8e5daa6cbec17e95f41debee7da2ff39c552fd806552\": container with ID starting with 2bf9ec4ecd66230932fc8e5daa6cbec17e95f41debee7da2ff39c552fd806552 not found: ID does not exist"
Jul 26 16:42:37.068450 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:37.068429    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bdfbca-5686c86bd4-csh8l"]
Jul 26 16:42:37.071610 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:37.071591    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bdfbca-5686c86bd4-csh8l"]
Jul 26 16:42:37.091986 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:37.091962    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bdfbca-5d9d85bd8b-7cl5p"]
Jul 26 16:42:37.101240 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:37.101217    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bdfbca-5d9d85bd8b-7cl5p"]
Jul 26 16:42:37.118526 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:37.118499    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bdfbca-5bfc8ffcd6-cwmtd"]
Jul 26 16:42:37.125799 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:37.125755    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bdfbca-5bfc8ffcd6-cwmtd"]
Jul 26 16:42:37.870381 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:37.870353    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:42:38.276387 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:38.276357    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:42:38.379474 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:38.379439    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4974c058-344e-4efc-8b4f-f33a05709acd" path="/var/lib/kubelet/pods/4974c058-344e-4efc-8b4f-f33a05709acd/volumes"
Jul 26 16:42:38.379839 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:38.379822    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9eab8b8-14ad-400e-9cb7-8a3a34ad3601" path="/var/lib/kubelet/pods/f9eab8b8-14ad-400e-9cb7-8a3a34ad3601/volumes"
Jul 26 16:42:38.380188 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:38.380176    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fefb5476-d140-46e0-87cf-128f5d761535" path="/var/lib/kubelet/pods/fefb5476-d140-46e0-87cf-128f5d761535/volumes"
Jul 26 16:42:43.357963 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:43.357926    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:42:51.758054 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:51.758014    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:42:51.860240 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:51.860179    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:42:59.361965 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:59.361929    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:42:59.462228 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:42:59.462189    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:43:20.388352 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:43:20.388287    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:43:21.548111 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:43:21.548079    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:43:27.996052 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:43:27.996018    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:43:28.094947 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:43:28.094914    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:43:28.407655 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:43:28.407575    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:43:28.583531 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:43:28.583501    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:43:41.111088 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:43:41.111058    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-33839d-58688b59b9-78zdw"]
Jul 26 16:43:41.111465 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:43:41.111263    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-33839d-58688b59b9-78zdw" podUID="779bfac4-792d-476b-8c4c-82638d8f53c5" containerName="maas-api" containerID="cri-o://a34e9fcdb93ee4a7f8141620eceedaa7c64d3bcd5406a9eae1c26718907e1332" gracePeriod=30
Jul 26 16:43:41.141580 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:43:41.141515    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:43:41.174303 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:43:41.174273    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-33839d-6b4fbc6664-sr92w"]
Jul 26 16:43:41.174526 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:43:41.174501    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-33839d-6b4fbc6664-sr92w" podUID="7901a20c-246a-42c4-a6d8-9fcaebb6edca" containerName="payload-pre-processing" containerID="cri-o://3b11dbf980b63cbcbafb81c6ed11be54de918fc77e72f16751eb2220f20183fe" gracePeriod=30
Jul 26 16:43:41.243095 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:43:41.243062    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:43:41.349301 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:43:41.349086    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:43:41.396087 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:43:41.395913    2563 generic.go:356] "Generic (PLEG): container finished" podID="779bfac4-792d-476b-8c4c-82638d8f53c5" containerID="a34e9fcdb93ee4a7f8141620eceedaa7c64d3bcd5406a9eae1c26718907e1332" exitCode=0
Jul 26 16:43:41.396087 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:43:41.395958    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-33839d-58688b59b9-78zdw" event={"ID":"779bfac4-792d-476b-8c4c-82638d8f53c5","Type":"ContainerDied","Data":"a34e9fcdb93ee4a7f8141620eceedaa7c64d3bcd5406a9eae1c26718907e1332"}
Jul 26 16:43:41.397942 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:43:41.397891    2563 generic.go:356] "Generic (PLEG): container finished" podID="7901a20c-246a-42c4-a6d8-9fcaebb6edca" containerID="3b11dbf980b63cbcbafb81c6ed11be54de918fc77e72f16751eb2220f20183fe" exitCode=0
Jul 26 16:43:41.398098 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:43:41.397953    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-33839d-6b4fbc6664-sr92w" event={"ID":"7901a20c-246a-42c4-a6d8-9fcaebb6edca","Type":"ContainerDied","Data":"3b11dbf980b63cbcbafb81c6ed11be54de918fc77e72f16751eb2220f20183fe"}
Jul 26 16:43:41.467913 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:43:41.467888    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-33839d-58688b59b9-78zdw"
Jul 26 16:43:41.486053 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:43:41.486027    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-33839d-6b4fbc6664-sr92w"
Jul 26 16:43:41.497494 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:43:41.497473    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w44tm\" (UniqueName: \"kubernetes.io/projected/779bfac4-792d-476b-8c4c-82638d8f53c5-kube-api-access-w44tm\") pod \"779bfac4-792d-476b-8c4c-82638d8f53c5\" (UID: \"779bfac4-792d-476b-8c4c-82638d8f53c5\") "
Jul 26 16:43:41.497617 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:43:41.497517    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/779bfac4-792d-476b-8c4c-82638d8f53c5-openshift-service-ca\") pod \"779bfac4-792d-476b-8c4c-82638d8f53c5\" (UID: \"779bfac4-792d-476b-8c4c-82638d8f53c5\") "
Jul 26 16:43:41.497617 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:43:41.497574    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/779bfac4-792d-476b-8c4c-82638d8f53c5-maas-api-tls\") pod \"779bfac4-792d-476b-8c4c-82638d8f53c5\" (UID: \"779bfac4-792d-476b-8c4c-82638d8f53c5\") "
Jul 26 16:43:41.497957 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:43:41.497923    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/779bfac4-792d-476b-8c4c-82638d8f53c5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "779bfac4-792d-476b-8c4c-82638d8f53c5" (UID: "779bfac4-792d-476b-8c4c-82638d8f53c5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 26 16:43:41.499569 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:43:41.499548    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/779bfac4-792d-476b-8c4c-82638d8f53c5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "779bfac4-792d-476b-8c4c-82638d8f53c5" (UID: "779bfac4-792d-476b-8c4c-82638d8f53c5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 26 16:43:41.499708 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:43:41.499684    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/779bfac4-792d-476b-8c4c-82638d8f53c5-kube-api-access-w44tm" (OuterVolumeSpecName: "kube-api-access-w44tm") pod "779bfac4-792d-476b-8c4c-82638d8f53c5" (UID: "779bfac4-792d-476b-8c4c-82638d8f53c5"). InnerVolumeSpecName "kube-api-access-w44tm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 26 16:43:41.598886 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:43:41.598847    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7901a20c-246a-42c4-a6d8-9fcaebb6edca-plugins-config-volume\") pod \"7901a20c-246a-42c4-a6d8-9fcaebb6edca\" (UID: \"7901a20c-246a-42c4-a6d8-9fcaebb6edca\") "
Jul 26 16:43:41.599040 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:43:41.598944    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-29c9l\" (UniqueName: \"kubernetes.io/projected/7901a20c-246a-42c4-a6d8-9fcaebb6edca-kube-api-access-29c9l\") pod \"7901a20c-246a-42c4-a6d8-9fcaebb6edca\" (UID: \"7901a20c-246a-42c4-a6d8-9fcaebb6edca\") "
Jul 26 16:43:41.599219 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:43:41.599202    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/779bfac4-792d-476b-8c4c-82638d8f53c5-maas-api-tls\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:43:41.599263 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:43:41.599226    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w44tm\" (UniqueName: \"kubernetes.io/projected/779bfac4-792d-476b-8c4c-82638d8f53c5-kube-api-access-w44tm\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:43:41.599263 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:43:41.599240    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/779bfac4-792d-476b-8c4c-82638d8f53c5-openshift-service-ca\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:43:41.599334 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:43:41.599293    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7901a20c-246a-42c4-a6d8-9fcaebb6edca-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7901a20c-246a-42c4-a6d8-9fcaebb6edca" (UID: "7901a20c-246a-42c4-a6d8-9fcaebb6edca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 26 16:43:41.601004 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:43:41.600987    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7901a20c-246a-42c4-a6d8-9fcaebb6edca-kube-api-access-29c9l" (OuterVolumeSpecName: "kube-api-access-29c9l") pod "7901a20c-246a-42c4-a6d8-9fcaebb6edca" (UID: "7901a20c-246a-42c4-a6d8-9fcaebb6edca"). InnerVolumeSpecName "kube-api-access-29c9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 26 16:43:41.700751 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:43:41.700674    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29c9l\" (UniqueName: \"kubernetes.io/projected/7901a20c-246a-42c4-a6d8-9fcaebb6edca-kube-api-access-29c9l\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:43:41.700751 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:43:41.700698    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7901a20c-246a-42c4-a6d8-9fcaebb6edca-plugins-config-volume\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:43:42.405092 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:43:42.405068    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-33839d-6b4fbc6664-sr92w"
Jul 26 16:43:42.405092 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:43:42.405084    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-33839d-6b4fbc6664-sr92w" event={"ID":"7901a20c-246a-42c4-a6d8-9fcaebb6edca","Type":"ContainerDied","Data":"aad1869127e210e04e4a96016fd60dff3fe0338c1b3f4fb19898388c09338dbd"}
Jul 26 16:43:42.405563 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:43:42.405145    2563 scope.go:117] "RemoveContainer" containerID="3b11dbf980b63cbcbafb81c6ed11be54de918fc77e72f16751eb2220f20183fe"
Jul 26 16:43:42.406510 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:43:42.406489    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-33839d-58688b59b9-78zdw" event={"ID":"779bfac4-792d-476b-8c4c-82638d8f53c5","Type":"ContainerDied","Data":"ec1284df3087f1d09bb7a12485af96c24331640a5f485cb2022df13a3833d0de"}
Jul 26 16:43:42.406570 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:43:42.406495    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-33839d-58688b59b9-78zdw"
Jul 26 16:43:42.414735 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:43:42.414705    2563 scope.go:117] "RemoveContainer" containerID="a34e9fcdb93ee4a7f8141620eceedaa7c64d3bcd5406a9eae1c26718907e1332"
Jul 26 16:43:42.427619 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:43:42.427594    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-33839d-58688b59b9-78zdw"]
Jul 26 16:43:42.433254 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:43:42.433231    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-33839d-58688b59b9-78zdw"]
Jul 26 16:43:42.448468 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:43:42.448443    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-33839d-6b4fbc6664-sr92w"]
Jul 26 16:43:42.452692 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:43:42.452672    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-33839d-6b4fbc6664-sr92w"]
Jul 26 16:43:43.855327 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:43:43.855283    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:43:44.251388 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:43:44.251358    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:43:44.378244 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:43:44.378204    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="779bfac4-792d-476b-8c4c-82638d8f53c5" path="/var/lib/kubelet/pods/779bfac4-792d-476b-8c4c-82638d8f53c5/volumes"
Jul 26 16:43:44.378833 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:43:44.378807    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7901a20c-246a-42c4-a6d8-9fcaebb6edca" path="/var/lib/kubelet/pods/7901a20c-246a-42c4-a6d8-9fcaebb6edca/volumes"
Jul 26 16:43:50.924415 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:43:50.924377    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:43:51.024365 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:43:51.024335    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:43:51.209355 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:43:51.209261    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:43:51.451774 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:43:51.451731    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:44:02.381540 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:02.381470    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-76272d-5b6cb9986b-88n5f"]
Jul 26 16:44:02.381984 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:02.381661    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-76272d-5b6cb9986b-88n5f" podUID="28f0c6d3-1b43-4dca-a4d7-1a7e1d77f082" containerName="maas-api" containerID="cri-o://1374c914dd9f29e8adfb9bbed79f8165f9b82f9029f0304a83f949e3b59d7062" gracePeriod=30
Jul 26 16:44:02.404783 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:02.404746    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:44:02.453991 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:02.453956    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-76272d-785765d686-xmmq6"]
Jul 26 16:44:02.454196 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:02.454172    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-76272d-785765d686-xmmq6" podUID="60187994-88fd-4943-9cb8-5316c18157de" containerName="payload-processing" containerID="cri-o://0fda0355ee14af9ac8ca6d7aa3562e54c2f1b757b2afe48cc56babe35ffb72bf" gracePeriod=30
Jul 26 16:44:02.470258 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:02.470224    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-76272d-547fc4cdb8-wvgmm"]
Jul 26 16:44:02.470506 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:02.470476    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-76272d-547fc4cdb8-wvgmm" podUID="fd05518b-df0b-4078-9c51-1c74eec82cfa" containerName="payload-pre-processing" containerID="cri-o://62179c84212d79db85bd45f4bbddd62fd2e277915104e796445fdf97cffc3c1e" gracePeriod=30
Jul 26 16:44:02.501940 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:02.501912    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:44:02.516547 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:02.516516    2563 generic.go:356] "Generic (PLEG): container finished" podID="28f0c6d3-1b43-4dca-a4d7-1a7e1d77f082" containerID="1374c914dd9f29e8adfb9bbed79f8165f9b82f9029f0304a83f949e3b59d7062" exitCode=0
Jul 26 16:44:02.516713 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:02.516590    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-76272d-5b6cb9986b-88n5f" event={"ID":"28f0c6d3-1b43-4dca-a4d7-1a7e1d77f082","Type":"ContainerDied","Data":"1374c914dd9f29e8adfb9bbed79f8165f9b82f9029f0304a83f949e3b59d7062"}
Jul 26 16:44:02.603012 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:02.602983    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:44:02.765642 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:02.765614    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-76272d-785765d686-xmmq6"
Jul 26 16:44:02.773282 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:02.773253    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-76272d-5b6cb9986b-88n5f"
Jul 26 16:44:02.782225 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:02.782202    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-76272d-547fc4cdb8-wvgmm"
Jul 26 16:44:02.903809 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:02.903701    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd05518b-df0b-4078-9c51-1c74eec82cfa-plugins-config-volume\") pod \"fd05518b-df0b-4078-9c51-1c74eec82cfa\" (UID: \"fd05518b-df0b-4078-9c51-1c74eec82cfa\") "
Jul 26 16:44:02.903809 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:02.903749    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28f0c6d3-1b43-4dca-a4d7-1a7e1d77f082-maas-api-tls\") pod \"28f0c6d3-1b43-4dca-a4d7-1a7e1d77f082\" (UID: \"28f0c6d3-1b43-4dca-a4d7-1a7e1d77f082\") "
Jul 26 16:44:02.904007 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:02.903902    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nq9tl\" (UniqueName: \"kubernetes.io/projected/28f0c6d3-1b43-4dca-a4d7-1a7e1d77f082-kube-api-access-nq9tl\") pod \"28f0c6d3-1b43-4dca-a4d7-1a7e1d77f082\" (UID: \"28f0c6d3-1b43-4dca-a4d7-1a7e1d77f082\") "
Jul 26 16:44:02.904007 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:02.903969    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60187994-88fd-4943-9cb8-5316c18157de-plugins-config-volume\") pod \"60187994-88fd-4943-9cb8-5316c18157de\" (UID: \"60187994-88fd-4943-9cb8-5316c18157de\") "
Jul 26 16:44:02.904007 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:02.903996    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dfx5r\" (UniqueName: \"kubernetes.io/projected/60187994-88fd-4943-9cb8-5316c18157de-kube-api-access-dfx5r\") pod \"60187994-88fd-4943-9cb8-5316c18157de\" (UID: \"60187994-88fd-4943-9cb8-5316c18157de\") "
Jul 26 16:44:02.904133 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:02.904064    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28f0c6d3-1b43-4dca-a4d7-1a7e1d77f082-openshift-service-ca\") pod \"28f0c6d3-1b43-4dca-a4d7-1a7e1d77f082\" (UID: \"28f0c6d3-1b43-4dca-a4d7-1a7e1d77f082\") "
Jul 26 16:44:02.904133 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:02.904077    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd05518b-df0b-4078-9c51-1c74eec82cfa-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fd05518b-df0b-4078-9c51-1c74eec82cfa" (UID: "fd05518b-df0b-4078-9c51-1c74eec82cfa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 26 16:44:02.904133 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:02.904094    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7ks84\" (UniqueName: \"kubernetes.io/projected/fd05518b-df0b-4078-9c51-1c74eec82cfa-kube-api-access-7ks84\") pod \"fd05518b-df0b-4078-9c51-1c74eec82cfa\" (UID: \"fd05518b-df0b-4078-9c51-1c74eec82cfa\") "
Jul 26 16:44:02.904372 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:02.904352    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28f0c6d3-1b43-4dca-a4d7-1a7e1d77f082-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "28f0c6d3-1b43-4dca-a4d7-1a7e1d77f082" (UID: "28f0c6d3-1b43-4dca-a4d7-1a7e1d77f082"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 26 16:44:02.904472 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:02.904378    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60187994-88fd-4943-9cb8-5316c18157de-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "60187994-88fd-4943-9cb8-5316c18157de" (UID: "60187994-88fd-4943-9cb8-5316c18157de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 26 16:44:02.904472 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:02.904433    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd05518b-df0b-4078-9c51-1c74eec82cfa-plugins-config-volume\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:44:02.906104 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:02.906069    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28f0c6d3-1b43-4dca-a4d7-1a7e1d77f082-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "28f0c6d3-1b43-4dca-a4d7-1a7e1d77f082" (UID: "28f0c6d3-1b43-4dca-a4d7-1a7e1d77f082"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 26 16:44:02.906205 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:02.906159    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60187994-88fd-4943-9cb8-5316c18157de-kube-api-access-dfx5r" (OuterVolumeSpecName: "kube-api-access-dfx5r") pod "60187994-88fd-4943-9cb8-5316c18157de" (UID: "60187994-88fd-4943-9cb8-5316c18157de"). InnerVolumeSpecName "kube-api-access-dfx5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 26 16:44:02.906205 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:02.906180    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28f0c6d3-1b43-4dca-a4d7-1a7e1d77f082-kube-api-access-nq9tl" (OuterVolumeSpecName: "kube-api-access-nq9tl") pod "28f0c6d3-1b43-4dca-a4d7-1a7e1d77f082" (UID: "28f0c6d3-1b43-4dca-a4d7-1a7e1d77f082"). InnerVolumeSpecName "kube-api-access-nq9tl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 26 16:44:02.906284 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:02.906225    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd05518b-df0b-4078-9c51-1c74eec82cfa-kube-api-access-7ks84" (OuterVolumeSpecName: "kube-api-access-7ks84") pod "fd05518b-df0b-4078-9c51-1c74eec82cfa" (UID: "fd05518b-df0b-4078-9c51-1c74eec82cfa"). InnerVolumeSpecName "kube-api-access-7ks84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 26 16:44:03.005795 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:03.005749    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28f0c6d3-1b43-4dca-a4d7-1a7e1d77f082-openshift-service-ca\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:44:03.005795 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:03.005792    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ks84\" (UniqueName: \"kubernetes.io/projected/fd05518b-df0b-4078-9c51-1c74eec82cfa-kube-api-access-7ks84\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:44:03.005795 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:03.005803    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28f0c6d3-1b43-4dca-a4d7-1a7e1d77f082-maas-api-tls\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:44:03.006041 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:03.005812    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nq9tl\" (UniqueName: \"kubernetes.io/projected/28f0c6d3-1b43-4dca-a4d7-1a7e1d77f082-kube-api-access-nq9tl\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:44:03.006041 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:03.005820    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60187994-88fd-4943-9cb8-5316c18157de-plugins-config-volume\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:44:03.006041 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:03.005828    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfx5r\" (UniqueName: \"kubernetes.io/projected/60187994-88fd-4943-9cb8-5316c18157de-kube-api-access-dfx5r\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:44:03.522648 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:03.522617    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-76272d-5b6cb9986b-88n5f"
Jul 26 16:44:03.522648 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:03.522643    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-76272d-5b6cb9986b-88n5f" event={"ID":"28f0c6d3-1b43-4dca-a4d7-1a7e1d77f082","Type":"ContainerDied","Data":"23e5e779411046d1f4eb9ea44ea50e2245649585fedf0befa61c1daa3a703635"}
Jul 26 16:44:03.523205 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:03.522709    2563 scope.go:117] "RemoveContainer" containerID="1374c914dd9f29e8adfb9bbed79f8165f9b82f9029f0304a83f949e3b59d7062"
Jul 26 16:44:03.524178 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:03.524050    2563 generic.go:356] "Generic (PLEG): container finished" podID="60187994-88fd-4943-9cb8-5316c18157de" containerID="0fda0355ee14af9ac8ca6d7aa3562e54c2f1b757b2afe48cc56babe35ffb72bf" exitCode=0
Jul 26 16:44:03.524178 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:03.524080    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-76272d-785765d686-xmmq6"
Jul 26 16:44:03.524178 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:03.524132    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-76272d-785765d686-xmmq6" event={"ID":"60187994-88fd-4943-9cb8-5316c18157de","Type":"ContainerDied","Data":"0fda0355ee14af9ac8ca6d7aa3562e54c2f1b757b2afe48cc56babe35ffb72bf"}
Jul 26 16:44:03.524364 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:03.524202    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-76272d-785765d686-xmmq6" event={"ID":"60187994-88fd-4943-9cb8-5316c18157de","Type":"ContainerDied","Data":"eb80020814df5a11691204045f78bd04ee101b3477751a848876b9203c28828f"}
Jul 26 16:44:03.525690 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:03.525668    2563 generic.go:356] "Generic (PLEG): container finished" podID="fd05518b-df0b-4078-9c51-1c74eec82cfa" containerID="62179c84212d79db85bd45f4bbddd62fd2e277915104e796445fdf97cffc3c1e" exitCode=0
Jul 26 16:44:03.525895 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:03.525708    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-76272d-547fc4cdb8-wvgmm"
Jul 26 16:44:03.525895 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:03.525717    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-76272d-547fc4cdb8-wvgmm" event={"ID":"fd05518b-df0b-4078-9c51-1c74eec82cfa","Type":"ContainerDied","Data":"62179c84212d79db85bd45f4bbddd62fd2e277915104e796445fdf97cffc3c1e"}
Jul 26 16:44:03.525895 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:03.525792    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-76272d-547fc4cdb8-wvgmm" event={"ID":"fd05518b-df0b-4078-9c51-1c74eec82cfa","Type":"ContainerDied","Data":"16d32ed704ac42d575c43ce148eabd046de6534cdb3c88ad5861bcc00725dc95"}
Jul 26 16:44:03.534423 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:03.534403    2563 scope.go:117] "RemoveContainer" containerID="0fda0355ee14af9ac8ca6d7aa3562e54c2f1b757b2afe48cc56babe35ffb72bf"
Jul 26 16:44:03.546667 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:03.546650    2563 scope.go:117] "RemoveContainer" containerID="0fda0355ee14af9ac8ca6d7aa3562e54c2f1b757b2afe48cc56babe35ffb72bf"
Jul 26 16:44:03.546973 ip-10-0-133-204 kubenswrapper[2563]: E0726 16:44:03.546954    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fda0355ee14af9ac8ca6d7aa3562e54c2f1b757b2afe48cc56babe35ffb72bf\": container with ID starting with 0fda0355ee14af9ac8ca6d7aa3562e54c2f1b757b2afe48cc56babe35ffb72bf not found: ID does not exist" containerID="0fda0355ee14af9ac8ca6d7aa3562e54c2f1b757b2afe48cc56babe35ffb72bf"
Jul 26 16:44:03.547035 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:03.546981    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fda0355ee14af9ac8ca6d7aa3562e54c2f1b757b2afe48cc56babe35ffb72bf"} err="failed to get container status \"0fda0355ee14af9ac8ca6d7aa3562e54c2f1b757b2afe48cc56babe35ffb72bf\": rpc error: code = NotFound desc = could not find container \"0fda0355ee14af9ac8ca6d7aa3562e54c2f1b757b2afe48cc56babe35ffb72bf\": container with ID starting with 0fda0355ee14af9ac8ca6d7aa3562e54c2f1b757b2afe48cc56babe35ffb72bf not found: ID does not exist"
Jul 26 16:44:03.547035 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:03.546997    2563 scope.go:117] "RemoveContainer" containerID="62179c84212d79db85bd45f4bbddd62fd2e277915104e796445fdf97cffc3c1e"
Jul 26 16:44:03.554357 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:03.554339    2563 scope.go:117] "RemoveContainer" containerID="62179c84212d79db85bd45f4bbddd62fd2e277915104e796445fdf97cffc3c1e"
Jul 26 16:44:03.554613 ip-10-0-133-204 kubenswrapper[2563]: E0726 16:44:03.554593    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62179c84212d79db85bd45f4bbddd62fd2e277915104e796445fdf97cffc3c1e\": container with ID starting with 62179c84212d79db85bd45f4bbddd62fd2e277915104e796445fdf97cffc3c1e not found: ID does not exist" containerID="62179c84212d79db85bd45f4bbddd62fd2e277915104e796445fdf97cffc3c1e"
Jul 26 16:44:03.554693 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:03.554623    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62179c84212d79db85bd45f4bbddd62fd2e277915104e796445fdf97cffc3c1e"} err="failed to get container status \"62179c84212d79db85bd45f4bbddd62fd2e277915104e796445fdf97cffc3c1e\": rpc error: code = NotFound desc = could not find container \"62179c84212d79db85bd45f4bbddd62fd2e277915104e796445fdf97cffc3c1e\": container with ID starting with 62179c84212d79db85bd45f4bbddd62fd2e277915104e796445fdf97cffc3c1e not found: ID does not exist"
Jul 26 16:44:03.558391 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:03.558363    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-76272d-5b6cb9986b-88n5f"]
Jul 26 16:44:03.563966 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:03.563945    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-76272d-5b6cb9986b-88n5f"]
Jul 26 16:44:03.577029 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:03.577011    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-76272d-785765d686-xmmq6"]
Jul 26 16:44:03.593010 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:03.592986    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-76272d-785765d686-xmmq6"]
Jul 26 16:44:03.606065 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:03.606043    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-76272d-547fc4cdb8-wvgmm"]
Jul 26 16:44:03.613445 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:03.613422    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-76272d-547fc4cdb8-wvgmm"]
Jul 26 16:44:04.372818 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:04.372790    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28f0c6d3-1b43-4dca-a4d7-1a7e1d77f082" path="/var/lib/kubelet/pods/28f0c6d3-1b43-4dca-a4d7-1a7e1d77f082/volumes"
Jul 26 16:44:04.373208 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:04.373195    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60187994-88fd-4943-9cb8-5316c18157de" path="/var/lib/kubelet/pods/60187994-88fd-4943-9cb8-5316c18157de/volumes"
Jul 26 16:44:04.373553 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:04.373540    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd05518b-df0b-4078-9c51-1c74eec82cfa" path="/var/lib/kubelet/pods/fd05518b-df0b-4078-9c51-1c74eec82cfa/volumes"
Jul 26 16:44:06.808261 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:06.808233    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:44:07.908051 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:07.908022    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:44:14.672470 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:14.672431    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:44:14.770358 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:14.770323    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:44:14.993505 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:14.993475    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:44:15.197970 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:15.197934    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:44:26.296136 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:26.296099    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-99124e-587f6b5499-778wz"]
Jul 26 16:44:26.296679 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:26.296357    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-99124e-587f6b5499-778wz" podUID="ed4e99e6-92b1-49cb-a4f0-a4f354b37462" containerName="maas-api" containerID="cri-o://5558aab8a2d04c1473a4b64d05b4d2b7dc6a37d26efd4fb27b11b29d591bea5b" gracePeriod=30
Jul 26 16:44:26.382072 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:26.382037    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-99124e-59545f7886-6874r"]
Jul 26 16:44:26.382290 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:26.382262    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-99124e-59545f7886-6874r" podUID="0727f694-727c-400d-98ba-e4309db6dd2e" containerName="payload-processing" containerID="cri-o://eabdc8185e9d6af7a934a315cebf170a441ed087850789a420022117f400217f" gracePeriod=30
Jul 26 16:44:26.394296 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:26.394156    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-99124e-76bff4c976-59pcc"]
Jul 26 16:44:26.394871 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:26.394510    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-99124e-76bff4c976-59pcc" podUID="0b75076d-7307-4c59-8c10-bc84eda0450b" containerName="payload-pre-processing" containerID="cri-o://8c6eed874fb6f3201a74c27435a6bf68496926b7faf1967171ca7d9affd56753" gracePeriod=30
Jul 26 16:44:26.643621 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:26.643592    2563 generic.go:356] "Generic (PLEG): container finished" podID="0727f694-727c-400d-98ba-e4309db6dd2e" containerID="eabdc8185e9d6af7a934a315cebf170a441ed087850789a420022117f400217f" exitCode=0
Jul 26 16:44:26.643799 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:26.643672    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-99124e-59545f7886-6874r" event={"ID":"0727f694-727c-400d-98ba-e4309db6dd2e","Type":"ContainerDied","Data":"eabdc8185e9d6af7a934a315cebf170a441ed087850789a420022117f400217f"}
Jul 26 16:44:26.645373 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:26.645349    2563 generic.go:356] "Generic (PLEG): container finished" podID="ed4e99e6-92b1-49cb-a4f0-a4f354b37462" containerID="5558aab8a2d04c1473a4b64d05b4d2b7dc6a37d26efd4fb27b11b29d591bea5b" exitCode=0
Jul 26 16:44:26.645488 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:26.645427    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-99124e-587f6b5499-778wz" event={"ID":"ed4e99e6-92b1-49cb-a4f0-a4f354b37462","Type":"ContainerDied","Data":"5558aab8a2d04c1473a4b64d05b4d2b7dc6a37d26efd4fb27b11b29d591bea5b"}
Jul 26 16:44:26.645488 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:26.645483    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-99124e-587f6b5499-778wz" event={"ID":"ed4e99e6-92b1-49cb-a4f0-a4f354b37462","Type":"ContainerDied","Data":"c54392f7b86e6fa9b2542ba98a499a8676b7c1aaafa4e468a7166ec171d8be64"}
Jul 26 16:44:26.645617 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:26.645509    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c54392f7b86e6fa9b2542ba98a499a8676b7c1aaafa4e468a7166ec171d8be64"
Jul 26 16:44:26.647149 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:26.647131    2563 generic.go:356] "Generic (PLEG): container finished" podID="0b75076d-7307-4c59-8c10-bc84eda0450b" containerID="8c6eed874fb6f3201a74c27435a6bf68496926b7faf1967171ca7d9affd56753" exitCode=0
Jul 26 16:44:26.647271 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:26.647152    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-99124e-76bff4c976-59pcc" event={"ID":"0b75076d-7307-4c59-8c10-bc84eda0450b","Type":"ContainerDied","Data":"8c6eed874fb6f3201a74c27435a6bf68496926b7faf1967171ca7d9affd56753"}
Jul 26 16:44:26.652262 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:26.652246    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-99124e-587f6b5499-778wz"
Jul 26 16:44:26.687038 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:26.686236    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-99124e-59545f7886-6874r"
Jul 26 16:44:26.719872 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:26.719847    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-99124e-76bff4c976-59pcc"
Jul 26 16:44:26.729171 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:26.729148    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed4e99e6-92b1-49cb-a4f0-a4f354b37462-openshift-service-ca\") pod \"ed4e99e6-92b1-49cb-a4f0-a4f354b37462\" (UID: \"ed4e99e6-92b1-49cb-a4f0-a4f354b37462\") "
Jul 26 16:44:26.729306 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:26.729204    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c4vn8\" (UniqueName: \"kubernetes.io/projected/0727f694-727c-400d-98ba-e4309db6dd2e-kube-api-access-c4vn8\") pod \"0727f694-727c-400d-98ba-e4309db6dd2e\" (UID: \"0727f694-727c-400d-98ba-e4309db6dd2e\") "
Jul 26 16:44:26.729306 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:26.729227    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed4e99e6-92b1-49cb-a4f0-a4f354b37462-maas-api-tls\") pod \"ed4e99e6-92b1-49cb-a4f0-a4f354b37462\" (UID: \"ed4e99e6-92b1-49cb-a4f0-a4f354b37462\") "
Jul 26 16:44:26.729306 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:26.729242    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0727f694-727c-400d-98ba-e4309db6dd2e-plugins-config-volume\") pod \"0727f694-727c-400d-98ba-e4309db6dd2e\" (UID: \"0727f694-727c-400d-98ba-e4309db6dd2e\") "
Jul 26 16:44:26.729306 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:26.729277    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rlbbl\" (UniqueName: \"kubernetes.io/projected/ed4e99e6-92b1-49cb-a4f0-a4f354b37462-kube-api-access-rlbbl\") pod \"ed4e99e6-92b1-49cb-a4f0-a4f354b37462\" (UID: \"ed4e99e6-92b1-49cb-a4f0-a4f354b37462\") "
Jul 26 16:44:26.729635 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:26.729607    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed4e99e6-92b1-49cb-a4f0-a4f354b37462-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ed4e99e6-92b1-49cb-a4f0-a4f354b37462" (UID: "ed4e99e6-92b1-49cb-a4f0-a4f354b37462"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 26 16:44:26.729721 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:26.729675    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0727f694-727c-400d-98ba-e4309db6dd2e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0727f694-727c-400d-98ba-e4309db6dd2e" (UID: "0727f694-727c-400d-98ba-e4309db6dd2e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 26 16:44:26.731470 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:26.731389    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0727f694-727c-400d-98ba-e4309db6dd2e-kube-api-access-c4vn8" (OuterVolumeSpecName: "kube-api-access-c4vn8") pod "0727f694-727c-400d-98ba-e4309db6dd2e" (UID: "0727f694-727c-400d-98ba-e4309db6dd2e"). InnerVolumeSpecName "kube-api-access-c4vn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 26 16:44:26.731579 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:26.731493    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed4e99e6-92b1-49cb-a4f0-a4f354b37462-kube-api-access-rlbbl" (OuterVolumeSpecName: "kube-api-access-rlbbl") pod "ed4e99e6-92b1-49cb-a4f0-a4f354b37462" (UID: "ed4e99e6-92b1-49cb-a4f0-a4f354b37462"). InnerVolumeSpecName "kube-api-access-rlbbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 26 16:44:26.731579 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:26.731485    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed4e99e6-92b1-49cb-a4f0-a4f354b37462-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ed4e99e6-92b1-49cb-a4f0-a4f354b37462" (UID: "ed4e99e6-92b1-49cb-a4f0-a4f354b37462"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 26 16:44:26.830468 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:26.830380    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qqz2c\" (UniqueName: \"kubernetes.io/projected/0b75076d-7307-4c59-8c10-bc84eda0450b-kube-api-access-qqz2c\") pod \"0b75076d-7307-4c59-8c10-bc84eda0450b\" (UID: \"0b75076d-7307-4c59-8c10-bc84eda0450b\") "
Jul 26 16:44:26.830468 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:26.830414    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b75076d-7307-4c59-8c10-bc84eda0450b-plugins-config-volume\") pod \"0b75076d-7307-4c59-8c10-bc84eda0450b\" (UID: \"0b75076d-7307-4c59-8c10-bc84eda0450b\") "
Jul 26 16:44:26.830661 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:26.830634    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed4e99e6-92b1-49cb-a4f0-a4f354b37462-openshift-service-ca\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:44:26.830661 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:26.830646    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4vn8\" (UniqueName: \"kubernetes.io/projected/0727f694-727c-400d-98ba-e4309db6dd2e-kube-api-access-c4vn8\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:44:26.830661 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:26.830655    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed4e99e6-92b1-49cb-a4f0-a4f354b37462-maas-api-tls\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:44:26.830798 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:26.830664    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0727f694-727c-400d-98ba-e4309db6dd2e-plugins-config-volume\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:44:26.830798 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:26.830673    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlbbl\" (UniqueName: \"kubernetes.io/projected/ed4e99e6-92b1-49cb-a4f0-a4f354b37462-kube-api-access-rlbbl\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:44:26.830873 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:26.830848    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b75076d-7307-4c59-8c10-bc84eda0450b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0b75076d-7307-4c59-8c10-bc84eda0450b" (UID: "0b75076d-7307-4c59-8c10-bc84eda0450b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 26 16:44:26.832470 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:26.832439    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b75076d-7307-4c59-8c10-bc84eda0450b-kube-api-access-qqz2c" (OuterVolumeSpecName: "kube-api-access-qqz2c") pod "0b75076d-7307-4c59-8c10-bc84eda0450b" (UID: "0b75076d-7307-4c59-8c10-bc84eda0450b"). InnerVolumeSpecName "kube-api-access-qqz2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 26 16:44:26.931269 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:26.931232    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b75076d-7307-4c59-8c10-bc84eda0450b-plugins-config-volume\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:44:26.931269 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:26.931260    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqz2c\" (UniqueName: \"kubernetes.io/projected/0b75076d-7307-4c59-8c10-bc84eda0450b-kube-api-access-qqz2c\") on node \"ip-10-0-133-204.ec2.internal\" DevicePath \"\""
Jul 26 16:44:27.001872 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:27.001823    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:44:27.101654 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:27.101562    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:44:27.653054 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:27.653024    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-99124e-76bff4c976-59pcc"
Jul 26 16:44:27.653488 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:27.653056    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-99124e-76bff4c976-59pcc" event={"ID":"0b75076d-7307-4c59-8c10-bc84eda0450b","Type":"ContainerDied","Data":"d6bd95bfdf90b4673fa482e533ddd79f272226cd52ae3080e2a2113b8933615f"}
Jul 26 16:44:27.653488 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:27.653124    2563 scope.go:117] "RemoveContainer" containerID="8c6eed874fb6f3201a74c27435a6bf68496926b7faf1967171ca7d9affd56753"
Jul 26 16:44:27.654462 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:27.654412    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-99124e-59545f7886-6874r"
Jul 26 16:44:27.654462 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:27.654433    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-99124e-587f6b5499-778wz"
Jul 26 16:44:27.654462 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:27.654452    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-99124e-59545f7886-6874r" event={"ID":"0727f694-727c-400d-98ba-e4309db6dd2e","Type":"ContainerDied","Data":"88b0a8810d1921e46d05833ddbc07ae4d441efb8215e6b2fd73e5002f18409af"}
Jul 26 16:44:27.661987 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:27.661967    2563 scope.go:117] "RemoveContainer" containerID="eabdc8185e9d6af7a934a315cebf170a441ed087850789a420022117f400217f"
Jul 26 16:44:27.712224 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:27.712193    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-99124e-587f6b5499-778wz"]
Jul 26 16:44:27.724458 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:27.724431    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-99124e-587f6b5499-778wz"]
Jul 26 16:44:27.752285 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:27.752253    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-99124e-76bff4c976-59pcc"]
Jul 26 16:44:27.759303 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:27.759277    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-99124e-76bff4c976-59pcc"]
Jul 26 16:44:27.785585 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:27.785558    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-99124e-59545f7886-6874r"]
Jul 26 16:44:27.794540 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:27.794513    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-99124e-59545f7886-6874r"]
Jul 26 16:44:28.373129 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:28.373089    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0727f694-727c-400d-98ba-e4309db6dd2e" path="/var/lib/kubelet/pods/0727f694-727c-400d-98ba-e4309db6dd2e/volumes"
Jul 26 16:44:28.373552 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:28.373531    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b75076d-7307-4c59-8c10-bc84eda0450b" path="/var/lib/kubelet/pods/0b75076d-7307-4c59-8c10-bc84eda0450b/volumes"
Jul 26 16:44:28.373971 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:28.373952    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed4e99e6-92b1-49cb-a4f0-a4f354b37462" path="/var/lib/kubelet/pods/ed4e99e6-92b1-49cb-a4f0-a4f354b37462/volumes"
Jul 26 16:44:30.409577 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:30.409541    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:44:30.817673 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:30.817638    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:44:37.330043 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:37.330008    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:44:37.428001 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:37.427969    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:44:37.649409 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:37.649324    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:44:37.851897 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:37.851867    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:44:44.049079 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:44.049047    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-7p94c_9e359ea2-4eda-44ed-a0fc-f7885f2891ab/manager/0.log"
Jul 26 16:44:44.175490 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:44.175461    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-t5q6d_26ad813d-5c13-4977-ad9e-09342b76426e/manager/0.log"
Jul 26 16:44:44.309172 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:44.309087    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5fb95c8f7b-gbwfs_c625fd17-2520-4288-8f6c-efb9f6b9d79c/manager/0.log"
Jul 26 16:44:44.439873 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:44.439849    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-jdqzl_70f22506-ae20-4e4b-873f-0bf16997cff6/server/0.log"
Jul 26 16:44:44.570387 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:44.570311    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-lr2rv_b23f9014-7117-43d6-975d-4f4e945db935/manager/0.log"
Jul 26 16:44:44.874322 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:44.874234    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-745bd4f5cc-pxsh5_128af9ad-0db4-46f5-bee7-993eabdfc2f4/copy-manifests/0.log"
Jul 26 16:44:44.893203 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:44.893179    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-745bd4f5cc-pxsh5_128af9ad-0db4-46f5-bee7-993eabdfc2f4/manager/0.log"
Jul 26 16:44:45.452911 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:45.452880    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-849dbfd774-dj2wk_08a0697f-7477-4170-9e68-f32144f104b4/maas-api/0.log"
Jul 26 16:44:45.936091 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:45.936061    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-g6vcc_f6d59668-5579-45c0-a659-3712c99a6fe3/postgres/0.log"
Jul 26 16:44:47.973511 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:47.973480    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-54d5v_4e276424-a20b-4eaa-b908-6d17df7bfcb3/manager/0.log"
Jul 26 16:44:48.740031 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:48.740001    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-5mp5d_9d5f0889-c6a8-4641-b6f4-28e81a9a7955/discovery/0.log"
Jul 26 16:44:48.864125 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:48.864090    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5bfd866646-9w5w8_c68b8fda-d0c9-479a-a3ed-dbef655aa0f7/kube-auth-proxy/0.log"
Jul 26 16:44:49.254860 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:49.254829    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-dhspm_f0863e78-2d3f-482a-bc97-6738e35a6618/payload-pre-processing/0.log"
Jul 26 16:44:49.394617 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:49.394568    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-rc9pz_9c08ac5a-85ff-4504-9d77-b8f41be0242a/payload-processing/0.log"
Jul 26 16:44:49.875992 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:49.875963    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-mlpjn_8862d5e5-d805-49f7-ac37-e6572367feac/main/0.log"
Jul 26 16:44:49.998508 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:49.998476    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-kvcvx_cde8a1d7-d445-4d0b-ba25-dc3e56d6657b/main/0.log"
Jul 26 16:44:56.651311 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:56.651269    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:44:56.801162 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:56.801115    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log" limit=1024
Jul 26 16:44:59.718517 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:59.718474    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-cxfp4_e62cc557-6267-4993-84a1-a888ced92345/global-pull-secret-syncer/0.log"
Jul 26 16:44:59.833432 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:44:59.833401    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-pj48f_0c48fa94-94f6-43da-af02-7d63be16a9b8/konnectivity-agent/0.log"
Jul 26 16:45:00.001337 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:00.001313    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-204.ec2.internal_6db9e0d726c4428a7173af2b853a8bd3/haproxy/0.log"
Jul 26 16:45:05.745744 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:05.745714    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-54d5v_4e276424-a20b-4eaa-b908-6d17df7bfcb3/manager/0.log"
Jul 26 16:45:07.474238 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:07.474212    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_32649266-3ac3-4f8d-8d02-0ea5b3462e41/alertmanager/0.log"
Jul 26 16:45:07.507002 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:07.506967    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_32649266-3ac3-4f8d-8d02-0ea5b3462e41/config-reloader/0.log"
Jul 26 16:45:07.541236 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:07.541216    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_32649266-3ac3-4f8d-8d02-0ea5b3462e41/kube-rbac-proxy-web/0.log"
Jul 26 16:45:07.570383 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:07.570361    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_32649266-3ac3-4f8d-8d02-0ea5b3462e41/kube-rbac-proxy/0.log"
Jul 26 16:45:07.596168 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:07.596147    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_32649266-3ac3-4f8d-8d02-0ea5b3462e41/kube-rbac-proxy-metric/0.log"
Jul 26 16:45:07.631879 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:07.631856    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_32649266-3ac3-4f8d-8d02-0ea5b3462e41/prom-label-proxy/0.log"
Jul 26 16:45:07.657034 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:07.657009    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_32649266-3ac3-4f8d-8d02-0ea5b3462e41/init-config-reloader/0.log"
Jul 26 16:45:07.860365 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:07.859030    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-94stp_5cd15409-1edd-41c2-bc75-1a5851945440/monitoring-plugin/0.log"
Jul 26 16:45:08.005609 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:08.005584    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pzvmc_0ed65756-c097-4851-9612-6a63e2cec317/node-exporter/0.log"
Jul 26 16:45:08.034527 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:08.034506    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pzvmc_0ed65756-c097-4851-9612-6a63e2cec317/kube-rbac-proxy/0.log"
Jul 26 16:45:08.063878 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:08.063854    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pzvmc_0ed65756-c097-4851-9612-6a63e2cec317/init-textfile/0.log"
Jul 26 16:45:08.307636 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:08.307608    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_645f034c-adf9-4a52-a059-2fa1d3988867/prometheus/0.log"
Jul 26 16:45:08.337349 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:08.337324    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_645f034c-adf9-4a52-a059-2fa1d3988867/config-reloader/0.log"
Jul 26 16:45:08.365352 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:08.365324    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_645f034c-adf9-4a52-a059-2fa1d3988867/thanos-sidecar/0.log"
Jul 26 16:45:08.396382 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:08.396361    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_645f034c-adf9-4a52-a059-2fa1d3988867/kube-rbac-proxy-web/0.log"
Jul 26 16:45:08.427781 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:08.427730    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_645f034c-adf9-4a52-a059-2fa1d3988867/kube-rbac-proxy/0.log"
Jul 26 16:45:08.455531 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:08.455501    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_645f034c-adf9-4a52-a059-2fa1d3988867/kube-rbac-proxy-thanos/0.log"
Jul 26 16:45:08.487268 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:08.487245    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_645f034c-adf9-4a52-a059-2fa1d3988867/init-config-reloader/0.log"
Jul 26 16:45:08.851000 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:08.850969    2563 scope.go:117] "RemoveContainer" containerID="5558aab8a2d04c1473a4b64d05b4d2b7dc6a37d26efd4fb27b11b29d591bea5b"
Jul 26 16:45:09.424520 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:09.424488    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5mkcz/perf-node-gather-daemonset-mx8wf"]
Jul 26 16:45:09.424885 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:09.424873    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28f0c6d3-1b43-4dca-a4d7-1a7e1d77f082" containerName="maas-api"
Jul 26 16:45:09.424941 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:09.424886    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="28f0c6d3-1b43-4dca-a4d7-1a7e1d77f082" containerName="maas-api"
Jul 26 16:45:09.424941 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:09.424895    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7901a20c-246a-42c4-a6d8-9fcaebb6edca" containerName="payload-pre-processing"
Jul 26 16:45:09.424941 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:09.424901    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7901a20c-246a-42c4-a6d8-9fcaebb6edca" containerName="payload-pre-processing"
Jul 26 16:45:09.424941 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:09.424908    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20faa6e9-3a39-422e-9742-ef1360793e79" containerName="maas-api"
Jul 26 16:45:09.424941 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:09.424913    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="20faa6e9-3a39-422e-9742-ef1360793e79" containerName="maas-api"
Jul 26 16:45:09.424941 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:09.424923    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fefb5476-d140-46e0-87cf-128f5d761535" containerName="maas-api"
Jul 26 16:45:09.424941 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:09.424929    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fefb5476-d140-46e0-87cf-128f5d761535" containerName="maas-api"
Jul 26 16:45:09.424941 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:09.424935    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="779bfac4-792d-476b-8c4c-82638d8f53c5" containerName="maas-api"
Jul 26 16:45:09.424941 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:09.424940    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="779bfac4-792d-476b-8c4c-82638d8f53c5" containerName="maas-api"
Jul 26 16:45:09.425208 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:09.424950    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4974c058-344e-4efc-8b4f-f33a05709acd" containerName="payload-processing"
Jul 26 16:45:09.425208 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:09.424955    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4974c058-344e-4efc-8b4f-f33a05709acd" containerName="payload-processing"
Jul 26 16:45:09.425208 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:09.424963    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0727f694-727c-400d-98ba-e4309db6dd2e" containerName="payload-processing"
Jul 26 16:45:09.425208 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:09.424968    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0727f694-727c-400d-98ba-e4309db6dd2e" containerName="payload-processing"
Jul 26 16:45:09.425208 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:09.424982    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed4e99e6-92b1-49cb-a4f0-a4f354b37462" containerName="maas-api"
Jul 26 16:45:09.425208 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:09.424989    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed4e99e6-92b1-49cb-a4f0-a4f354b37462" containerName="maas-api"
Jul 26 16:45:09.425208 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:09.424996    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd05518b-df0b-4078-9c51-1c74eec82cfa" containerName="payload-pre-processing"
Jul 26 16:45:09.425208 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:09.425001    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd05518b-df0b-4078-9c51-1c74eec82cfa" containerName="payload-pre-processing"
Jul 26 16:45:09.425208 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:09.425009    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60187994-88fd-4943-9cb8-5316c18157de" containerName="payload-processing"
Jul 26 16:45:09.425208 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:09.425013    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="60187994-88fd-4943-9cb8-5316c18157de" containerName="payload-processing"
Jul 26 16:45:09.425208 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:09.425066    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b75076d-7307-4c59-8c10-bc84eda0450b" containerName="payload-pre-processing"
Jul 26 16:45:09.425208 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:09.425074    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b75076d-7307-4c59-8c10-bc84eda0450b" containerName="payload-pre-processing"
Jul 26 16:45:09.425208 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:09.425103    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9eab8b8-14ad-400e-9cb7-8a3a34ad3601" containerName="payload-pre-processing"
Jul 26 16:45:09.425208 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:09.425111    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9eab8b8-14ad-400e-9cb7-8a3a34ad3601" containerName="payload-pre-processing"
Jul 26 16:45:09.425208 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:09.425190    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7901a20c-246a-42c4-a6d8-9fcaebb6edca" containerName="payload-pre-processing"
Jul 26 16:45:09.425208 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:09.425205    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="779bfac4-792d-476b-8c4c-82638d8f53c5" containerName="maas-api"
Jul 26 16:45:09.425208 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:09.425212    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed4e99e6-92b1-49cb-a4f0-a4f354b37462" containerName="maas-api"
Jul 26 16:45:09.425720 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:09.425217    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="28f0c6d3-1b43-4dca-a4d7-1a7e1d77f082" containerName="maas-api"
Jul 26 16:45:09.425720 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:09.425225    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0727f694-727c-400d-98ba-e4309db6dd2e" containerName="payload-processing"
Jul 26 16:45:09.425720 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:09.425231    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4974c058-344e-4efc-8b4f-f33a05709acd" containerName="payload-processing"
Jul 26 16:45:09.425720 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:09.425236    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="20faa6e9-3a39-422e-9742-ef1360793e79" containerName="maas-api"
Jul 26 16:45:09.425720 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:09.425241    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b75076d-7307-4c59-8c10-bc84eda0450b" containerName="payload-pre-processing"
Jul 26 16:45:09.425720 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:09.425248    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd05518b-df0b-4078-9c51-1c74eec82cfa" containerName="payload-pre-processing"
Jul 26 16:45:09.425720 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:09.425255    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fefb5476-d140-46e0-87cf-128f5d761535" containerName="maas-api"
Jul 26 16:45:09.425720 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:09.425266    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="60187994-88fd-4943-9cb8-5316c18157de" containerName="payload-processing"
Jul 26 16:45:09.425720 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:09.425273    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9eab8b8-14ad-400e-9cb7-8a3a34ad3601" containerName="payload-pre-processing"
Jul 26 16:45:09.429845 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:09.429821    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5mkcz/perf-node-gather-daemonset-mx8wf"
Jul 26 16:45:09.432084 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:09.432066    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5mkcz\"/\"kube-root-ca.crt\""
Jul 26 16:45:09.432868 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:09.432851    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5mkcz\"/\"openshift-service-ca.crt\""
Jul 26 16:45:09.432919 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:09.432887    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5mkcz\"/\"default-dockercfg-bdkcf\""
Jul 26 16:45:09.441566 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:09.441546    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5mkcz/perf-node-gather-daemonset-mx8wf"]
Jul 26 16:45:09.520067 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:09.520029    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cfc8174b-5d37-43b1-aa40-039102df76b4-proc\") pod \"perf-node-gather-daemonset-mx8wf\" (UID: \"cfc8174b-5d37-43b1-aa40-039102df76b4\") " pod="openshift-must-gather-5mkcz/perf-node-gather-daemonset-mx8wf"
Jul 26 16:45:09.520528 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:09.520153    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cfc8174b-5d37-43b1-aa40-039102df76b4-podres\") pod \"perf-node-gather-daemonset-mx8wf\" (UID: \"cfc8174b-5d37-43b1-aa40-039102df76b4\") " pod="openshift-must-gather-5mkcz/perf-node-gather-daemonset-mx8wf"
Jul 26 16:45:09.520528 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:09.520226    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cfc8174b-5d37-43b1-aa40-039102df76b4-sys\") pod \"perf-node-gather-daemonset-mx8wf\" (UID: \"cfc8174b-5d37-43b1-aa40-039102df76b4\") " pod="openshift-must-gather-5mkcz/perf-node-gather-daemonset-mx8wf"
Jul 26 16:45:09.520528 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:09.520280    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srf57\" (UniqueName: \"kubernetes.io/projected/cfc8174b-5d37-43b1-aa40-039102df76b4-kube-api-access-srf57\") pod \"perf-node-gather-daemonset-mx8wf\" (UID: \"cfc8174b-5d37-43b1-aa40-039102df76b4\") " pod="openshift-must-gather-5mkcz/perf-node-gather-daemonset-mx8wf"
Jul 26 16:45:09.520528 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:09.520307    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cfc8174b-5d37-43b1-aa40-039102df76b4-lib-modules\") pod \"perf-node-gather-daemonset-mx8wf\" (UID: \"cfc8174b-5d37-43b1-aa40-039102df76b4\") " pod="openshift-must-gather-5mkcz/perf-node-gather-daemonset-mx8wf"
Jul 26 16:45:09.620895 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:09.620859    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cfc8174b-5d37-43b1-aa40-039102df76b4-podres\") pod \"perf-node-gather-daemonset-mx8wf\" (UID: \"cfc8174b-5d37-43b1-aa40-039102df76b4\") " pod="openshift-must-gather-5mkcz/perf-node-gather-daemonset-mx8wf"
Jul 26 16:45:09.621092 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:09.620912    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cfc8174b-5d37-43b1-aa40-039102df76b4-sys\") pod \"perf-node-gather-daemonset-mx8wf\" (UID: \"cfc8174b-5d37-43b1-aa40-039102df76b4\") " pod="openshift-must-gather-5mkcz/perf-node-gather-daemonset-mx8wf"
Jul 26 16:45:09.621092 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:09.620957    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srf57\" (UniqueName: \"kubernetes.io/projected/cfc8174b-5d37-43b1-aa40-039102df76b4-kube-api-access-srf57\") pod \"perf-node-gather-daemonset-mx8wf\" (UID: \"cfc8174b-5d37-43b1-aa40-039102df76b4\") " pod="openshift-must-gather-5mkcz/perf-node-gather-daemonset-mx8wf"
Jul 26 16:45:09.621092 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:09.620979    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cfc8174b-5d37-43b1-aa40-039102df76b4-lib-modules\") pod \"perf-node-gather-daemonset-mx8wf\" (UID: \"cfc8174b-5d37-43b1-aa40-039102df76b4\") " pod="openshift-must-gather-5mkcz/perf-node-gather-daemonset-mx8wf"
Jul 26 16:45:09.621092 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:09.621006    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cfc8174b-5d37-43b1-aa40-039102df76b4-proc\") pod \"perf-node-gather-daemonset-mx8wf\" (UID: \"cfc8174b-5d37-43b1-aa40-039102df76b4\") " pod="openshift-must-gather-5mkcz/perf-node-gather-daemonset-mx8wf"
Jul 26 16:45:09.621092 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:09.621050    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cfc8174b-5d37-43b1-aa40-039102df76b4-podres\") pod \"perf-node-gather-daemonset-mx8wf\" (UID: \"cfc8174b-5d37-43b1-aa40-039102df76b4\") " pod="openshift-must-gather-5mkcz/perf-node-gather-daemonset-mx8wf"
Jul 26 16:45:09.621092 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:09.621073    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cfc8174b-5d37-43b1-aa40-039102df76b4-sys\") pod \"perf-node-gather-daemonset-mx8wf\" (UID: \"cfc8174b-5d37-43b1-aa40-039102df76b4\") " pod="openshift-must-gather-5mkcz/perf-node-gather-daemonset-mx8wf"
Jul 26 16:45:09.621322 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:09.621107    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cfc8174b-5d37-43b1-aa40-039102df76b4-proc\") pod \"perf-node-gather-daemonset-mx8wf\" (UID: \"cfc8174b-5d37-43b1-aa40-039102df76b4\") " pod="openshift-must-gather-5mkcz/perf-node-gather-daemonset-mx8wf"
Jul 26 16:45:09.621322 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:09.621133    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cfc8174b-5d37-43b1-aa40-039102df76b4-lib-modules\") pod \"perf-node-gather-daemonset-mx8wf\" (UID: \"cfc8174b-5d37-43b1-aa40-039102df76b4\") " pod="openshift-must-gather-5mkcz/perf-node-gather-daemonset-mx8wf"
Jul 26 16:45:09.639038 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:09.639013    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srf57\" (UniqueName: \"kubernetes.io/projected/cfc8174b-5d37-43b1-aa40-039102df76b4-kube-api-access-srf57\") pod \"perf-node-gather-daemonset-mx8wf\" (UID: \"cfc8174b-5d37-43b1-aa40-039102df76b4\") " pod="openshift-must-gather-5mkcz/perf-node-gather-daemonset-mx8wf"
Jul 26 16:45:09.739243 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:09.739217    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5mkcz/perf-node-gather-daemonset-mx8wf"
Jul 26 16:45:09.871845 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:09.871814    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5mkcz/perf-node-gather-daemonset-mx8wf"]
Jul 26 16:45:09.875675 ip-10-0-133-204 kubenswrapper[2563]: W0726 16:45:09.875644    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podcfc8174b_5d37_43b1_aa40_039102df76b4.slice/crio-57168f421e7f1f8ee90cd68a31da2d1d7fc163214086d8a74b2b7096975151a0 WatchSource:0}: Error finding container 57168f421e7f1f8ee90cd68a31da2d1d7fc163214086d8a74b2b7096975151a0: Status 404 returned error can't find the container with id 57168f421e7f1f8ee90cd68a31da2d1d7fc163214086d8a74b2b7096975151a0
Jul 26 16:45:10.861155 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:10.861119    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5mkcz/perf-node-gather-daemonset-mx8wf" event={"ID":"cfc8174b-5d37-43b1-aa40-039102df76b4","Type":"ContainerStarted","Data":"9fda44e7d321ee6abc8a8892feb29d954827f7aa796e7c8062af34909b330fdc"}
Jul 26 16:45:10.861553 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:10.861195    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5mkcz/perf-node-gather-daemonset-mx8wf" event={"ID":"cfc8174b-5d37-43b1-aa40-039102df76b4","Type":"ContainerStarted","Data":"57168f421e7f1f8ee90cd68a31da2d1d7fc163214086d8a74b2b7096975151a0"}
Jul 26 16:45:10.861553 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:10.861260    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5mkcz/perf-node-gather-daemonset-mx8wf"
Jul 26 16:45:10.885376 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:10.885319    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5mkcz/perf-node-gather-daemonset-mx8wf" podStartSLOduration=1.885304066 podStartE2EDuration="1.885304066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 16:45:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:45:10.883611237 +0000 UTC m=+3302.987837060" watchObservedRunningTime="2026-07-26 16:45:10.885304066 +0000 UTC m=+3302.989529874"
Jul 26 16:45:13.064816 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:13.064786    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9wvxg_7c34afa8-d2eb-4651-8464-4a56f7605447/dns/0.log"
Jul 26 16:45:13.123187 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:13.123162    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9wvxg_7c34afa8-d2eb-4651-8464-4a56f7605447/kube-rbac-proxy/0.log"
Jul 26 16:45:13.416931 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:13.416859    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gnkxb_aba3d036-e332-4b06-8379-7deb395345ba/dns-node-resolver/0.log"
Jul 26 16:45:14.155408 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:14.155377    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-p8kcf_e1f2fc7a-9737-422c-9c17-c8821dc48396/node-ca/0.log"
Jul 26 16:45:15.484335 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:15.484307    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-5mp5d_9d5f0889-c6a8-4641-b6f4-28e81a9a7955/discovery/0.log"
Jul 26 16:45:15.509044 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:15.509021    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5bfd866646-9w5w8_c68b8fda-d0c9-479a-a3ed-dbef655aa0f7/kube-auth-proxy/0.log"
Jul 26 16:45:15.641219 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:15.641174    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-dhspm_f0863e78-2d3f-482a-bc97-6738e35a6618/payload-pre-processing/0.log"
Jul 26 16:45:15.672259 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:15.672232    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-rc9pz_9c08ac5a-85ff-4504-9d77-b8f41be0242a/payload-processing/0.log"
Jul 26 16:45:16.535665 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:16.535637    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9mkmv_214a33cc-fbe9-4dd9-8f42-14e737ac0341/serve-healthcheck-canary/0.log"
Jul 26 16:45:16.875932 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:16.875863    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5mkcz/perf-node-gather-daemonset-mx8wf"
Jul 26 16:45:17.089858 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:17.089831    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-lgfrd_f135e47d-f8a0-413b-aed2-37d48bc1a2f7/insights-operator/0.log"
Jul 26 16:45:17.248259 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:17.248230    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6shks_212d7a6b-fb00-48bf-88a1-c5d617abe2cb/kube-rbac-proxy/0.log"
Jul 26 16:45:17.290007 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:17.289982    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6shks_212d7a6b-fb00-48bf-88a1-c5d617abe2cb/exporter/0.log"
Jul 26 16:45:17.330000 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:17.329975    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6shks_212d7a6b-fb00-48bf-88a1-c5d617abe2cb/extractor/0.log"
Jul 26 16:45:19.677166 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:19.677135    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-5mp5d_9d5f0889-c6a8-4641-b6f4-28e81a9a7955/discovery/0.log"
Jul 26 16:45:19.689504 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:19.689480    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5bfd866646-9w5w8_c68b8fda-d0c9-479a-a3ed-dbef655aa0f7/kube-auth-proxy/0.log"
Jul 26 16:45:19.792964 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:19.792931    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-dhspm_f0863e78-2d3f-482a-bc97-6738e35a6618/payload-pre-processing/0.log"
Jul 26 16:45:19.811871 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:19.811843    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-rc9pz_9c08ac5a-85ff-4504-9d77-b8f41be0242a/payload-processing/0.log"
Jul 26 16:45:19.865518 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:19.865444    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-7p94c_9e359ea2-4eda-44ed-a0fc-f7885f2891ab/manager/0.log"
Jul 26 16:45:20.073505 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:20.073467    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-t5q6d_26ad813d-5c13-4977-ad9e-09342b76426e/manager/0.log"
Jul 26 16:45:20.327635 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:20.327596    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5fb95c8f7b-gbwfs_c625fd17-2520-4288-8f6c-efb9f6b9d79c/manager/0.log"
Jul 26 16:45:20.351830 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:20.351797    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-jdqzl_70f22506-ae20-4e4b-873f-0bf16997cff6/server/0.log"
Jul 26 16:45:20.442185 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:20.442138    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-lr2rv_b23f9014-7117-43d6-975d-4f4e945db935/manager/0.log"
Jul 26 16:45:20.598389 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:20.598310    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-745bd4f5cc-pxsh5_128af9ad-0db4-46f5-bee7-993eabdfc2f4/manager/0.log"
Jul 26 16:45:20.618629 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:20.618604    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-745bd4f5cc-pxsh5_128af9ad-0db4-46f5-bee7-993eabdfc2f4/copy-manifests/0.log"
Jul 26 16:45:21.455849 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:21.455818    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-5mp5d_9d5f0889-c6a8-4641-b6f4-28e81a9a7955/discovery/0.log"
Jul 26 16:45:21.473066 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:21.473045    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5bfd866646-9w5w8_c68b8fda-d0c9-479a-a3ed-dbef655aa0f7/kube-auth-proxy/0.log"
Jul 26 16:45:21.573379 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:21.573347    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-dhspm_f0863e78-2d3f-482a-bc97-6738e35a6618/payload-pre-processing/0.log"
Jul 26 16:45:21.594091 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:21.594069    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-rc9pz_9c08ac5a-85ff-4504-9d77-b8f41be0242a/payload-processing/0.log"
Jul 26 16:45:26.909682 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:26.909653    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-6fgd7_d572adcb-06e0-4f95-9de7-8d72b2428789/migrator/0.log"
Jul 26 16:45:26.937353 ip-10-0-133-204 kubenswrapper[2563]: I0726 16:45:26.937325    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-6fgd7_d572adcb-06e0-4f95-9de7-8d72b2428789/graceful-termination/0.log"